Sep 10 12:30:33.267858 ip-10-0-137-89 systemd[1]: Starting Kubernetes Kubelet... Sep 10 12:30:33.626070 ip-10-0-137-89 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 12:30:33.626070 ip-10-0-137-89 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 12:30:33.626070 ip-10-0-137-89 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 12:30:33.626070 ip-10-0-137-89 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 12:30:33.626070 ip-10-0-137-89 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 12:30:33.628120 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.628011 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 12:30:33.631679 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631664 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 12:30:33.631679 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631679 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631683 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631686 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631689 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631692 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631696 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631698 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631701 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631703 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631706 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631709 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631711 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631714 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631717 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631720 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631722 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631724 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631727 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631730 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631732 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 12:30:33.631742 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631735 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631738 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631742 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631746 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631749 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631752 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631755 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631757 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631760 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631762 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631765 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631767 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631770 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631772 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631775 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631777 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631781 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631784 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631787 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 12:30:33.632250 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631791 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631793 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631796 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631798 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631801 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631803 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631805 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631808 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631810 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631813 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631815 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631817 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631821 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631823 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631826 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631829 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631832 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631835 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631837 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631840 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 12:30:33.632725 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631842 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631845 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631848 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631851 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631854 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631857 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631860 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631863 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631866 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631868 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631871 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631874 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631876 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631879 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631882 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631884 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631887 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631890 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631892 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 12:30:33.633219 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631895 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631899 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631902 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631904 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631906 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631909 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631911 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631914 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631917 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631920 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631923 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631925 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631928 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631930 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631933 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631935 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631937 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631940 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631942 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 12:30:33.633692 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.631945 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632617 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632624 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632627 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632629 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632632 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632635 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632637 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632640 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632642 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632645 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632648 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632651 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632653 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632656 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632659 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632662 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632665 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632668 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 12:30:33.634141 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632672 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632676 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632680 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632683 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632686 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632689 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632692 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632694 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632697 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632700 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632702 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632704 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632707 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632709 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632712 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632714 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632717 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632719 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632721 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632724 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 12:30:33.634621 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632726 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632729 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632732 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632735 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632737 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632740 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632743 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632745 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632748 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632751 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632753 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632756 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632758 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632760 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632764 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632766 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632769 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632771 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632774 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 12:30:33.635112 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632776 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632778 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632781 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632784 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632787 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632789 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632792 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632794 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632796 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632799 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632802 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632804 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632815 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632818 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632821 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632825 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632828 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632830 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632833 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632836 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 12:30:33.635577 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632838 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632841 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632843 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632846 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632848 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632851 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632855 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632859 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632863 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632866 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632870 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632872 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632875 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632878 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632880 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632883 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632886 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632888 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632891 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 12:30:33.636110 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632893 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632896 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.632899 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633267 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633272 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633275 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633278 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633281 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633283 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633286 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633289 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633291 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633296 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633299 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633303 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633307 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633310 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633313 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633316 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633319 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 12:30:33.636568 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633322 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633324 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633327 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633329 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633332 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633335 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633337 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633339 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633342 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633344 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633347 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633349 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633352 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633355 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633357 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633360 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633362 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633364 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633367 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633369 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 12:30:33.637075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633372 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633374 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633377 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633380 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633383 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633386 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633388 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633391 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633395 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633399 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633402 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633405 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633407 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633410 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633413 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633415 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633418 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633420 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633423 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633425 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 12:30:33.637573 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633428 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633430 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633433 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633435 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633438 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633440 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633443 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633445 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633447 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633450 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633452 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633454 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633457 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633459 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633462 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633464 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633471 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633475 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633478 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633480 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 12:30:33.638075 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633483 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633485 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633490 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633493 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633495 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633498 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633500 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633502 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633505 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633508 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633511 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633513 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633515 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633517 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633520 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633523 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633525 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633528 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633530 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 12:30:33.638555 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633533 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633535 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.633538 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633623 2562 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633665 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633675 2562 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633679 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633687 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633690 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633696 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633700 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633703 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633706 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633709 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633712 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633715 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633719 2562 flags.go:64] FLAG: --cgroup-root="" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633722 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633725 2562 flags.go:64] FLAG: --client-ca-file="" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633728 2562 flags.go:64] FLAG: --cloud-provider="external" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633731 2562 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633735 2562 flags.go:64] FLAG: --cluster-domain="" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633737 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633740 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 12:30:33.639077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633743 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633747 2562 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633750 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633753 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633756 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633760 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633763 2562 flags.go:64] FLAG: --contention-profiling="false" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633766 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633769 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633772 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633775 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633778 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633781 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633784 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633787 2562 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633790 2562 flags.go:64] FLAG: --enable-server="true" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633792 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633797 2562 flags.go:64] FLAG: --event-burst="100" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633800 2562 flags.go:64] FLAG: --event-qps="50" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633803 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633805 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633808 2562 flags.go:64] FLAG: --eviction-hard="" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633812 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633815 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633818 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 12:30:33.639661 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633821 2562 flags.go:64] FLAG: --eviction-soft="" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633824 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633827 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633830 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633832 2562 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633835 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633838 2562 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633841 2562 flags.go:64] FLAG: --feature-gates="" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633844 2562 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633847 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633850 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633853 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633858 2562 flags.go:64] FLAG: --healthz-port="10248" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633861 2562 flags.go:64] FLAG: --help="false" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633864 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-137-89.ec2.internal" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633867 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633870 2562 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633873 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633876 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633879 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633882 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633885 2562 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633887 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 12:30:33.640244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633890 2562 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633893 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633896 2562 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633899 2562 flags.go:64] FLAG: --kube-reserved="" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633902 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633905 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633908 2562 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633910 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633913 2562 flags.go:64] FLAG: --lock-file="" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633916 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633919 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633922 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633927 2562 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633930 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633932 2562 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633935 2562 flags.go:64] FLAG: --logging-format="text" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633938 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633941 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633944 2562 flags.go:64] FLAG: --manifest-url="" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633946 2562 flags.go:64] FLAG: --manifest-url-header="" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633950 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633955 2562 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633959 2562 flags.go:64] FLAG: --max-pods="110" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633962 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633964 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 12:30:33.640845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633967 2562 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633970 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633973 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633975 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633978 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633985 2562 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633988 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633991 2562 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633994 2562 flags.go:64] FLAG: --pod-cidr="" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.633997 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634002 2562 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634005 2562 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634008 2562 flags.go:64] FLAG: --pods-per-core="0" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634011 2562 flags.go:64] FLAG: --port="10250" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634014 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634016 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0210d0e3dbfb572ce" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634019 2562 flags.go:64] FLAG: --qos-reserved="" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634022 2562 flags.go:64] FLAG: --read-only-port="10255" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634026 2562 flags.go:64] FLAG: --register-node="true" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634028 2562 flags.go:64] FLAG: --register-with-taints="" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634032 2562 flags.go:64] FLAG: --registry-burst="10" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634035 2562 flags.go:64] FLAG: --registry-qps="5" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634037 2562 flags.go:64] FLAG: --reserved-cpus="" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634040 2562 flags.go:64] FLAG: --reserved-memory="" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634044 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 12:30:33.641450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634047 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634050 2562 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634052 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634055 2562 flags.go:64] FLAG: --runonce="false" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634059 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634062 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634065 2562 flags.go:64] FLAG: --seccomp-default="false" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634068 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634071 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634074 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634077 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634080 2562 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634083 2562 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634085 2562 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634088 2562 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634091 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634094 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634097 2562 flags.go:64] FLAG: --system-cgroups="" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634100 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634105 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634108 2562 flags.go:64] FLAG: --tls-cert-file="" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634111 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634114 2562 flags.go:64] FLAG: --tls-min-version="" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634117 2562 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634119 2562 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 12:30:33.642066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634122 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634126 2562 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634128 2562 flags.go:64] FLAG: --v="2" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634132 2562 flags.go:64] FLAG: --version="false" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634135 2562 flags.go:64] FLAG: --vmodule="" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634139 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634142 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634251 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634255 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634257 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634260 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634264 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634267 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634270 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634273 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634276 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634278 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634281 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634284 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634286 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634288 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 12:30:33.642743 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634291 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634294 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634297 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634299 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634302 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634304 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634307 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634309 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634312 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634315 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634317 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634320 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634322 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634325 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634328 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634330 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634332 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634335 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634337 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634340 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 12:30:33.643256 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634342 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634345 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634348 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634351 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634353 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634356 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634359 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634361 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634364 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634366 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634369 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634371 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634373 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634377 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634379 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634382 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634384 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634388 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634401 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 12:30:33.643768 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634403 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634406 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634409 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634412 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634415 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634419 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634422 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634426 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634429 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634431 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634434 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634436 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634439 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634441 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634444 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634447 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634450 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634453 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634455 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 12:30:33.644227 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634459 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634462 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634465 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634468 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634470 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634473 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634476 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634479 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634482 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634484 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634487 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634489 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634491 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634494 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634496 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634499 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634501 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634504 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634506 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634509 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 12:30:33.644713 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634511 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634514 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634517 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634519 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634522 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634524 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.634526 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.634532 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.641705 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.641721 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.642505 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.642516 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642610 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642616 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 12:30:33.645201 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642619 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642623 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642626 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642628 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642632 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642635 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642637 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642640 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642644 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642648 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642651 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642654 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642657 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642659 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642662 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642665 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642669 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642672 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642675 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 12:30:33.645603 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642678 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642680 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642684 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642686 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642689 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642691 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642694 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642696 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642706 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642709 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642712 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642714 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642716 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642719 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642721 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642724 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642726 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642728 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642731 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 12:30:33.646146 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642733 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642736 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642738 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642741 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642743 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642746 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642748 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642751 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642754 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642756 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642759 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642761 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642764 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642766 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642768 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642771 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642773 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642776 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642778 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642781 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 12:30:33.646629 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642783 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642786 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642794 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642797 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642799 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642802 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642804 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642806 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642809 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642811 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642814 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642816 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642819 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642821 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642823 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642826 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642828 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642830 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642833 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642835 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 12:30:33.647106 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642837 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642840 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642842 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642845 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642847 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642850 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642853 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642855 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642858 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642860 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642862 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642865 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642867 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642870 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642872 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642881 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642885 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642889 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 12:30:33.647625 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.642892 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.642897 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643046 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643051 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643054 2562 feature_gate.go:328] unrecognized feature gate: Example Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643057 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643059 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643062 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643065 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643067 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643070 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643072 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643074 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643077 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643079 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 12:30:33.648053 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643083 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643087 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643090 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643092 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643095 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643099 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643101 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643104 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643106 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643109 2562 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643111 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643114 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643116 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643118 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643121 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643130 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643133 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643135 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643138 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643140 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 12:30:33.648412 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643143 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643145 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643147 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643150 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643152 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643154 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643157 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643160 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643164 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643167 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643169 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643171 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643174 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643176 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643179 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643181 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643184 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643187 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643189 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 12:30:33.649031 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643191 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643194 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643196 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643199 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643201 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643203 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643205 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643208 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643210 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643218 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643221 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643223 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643225 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643229 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643232 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643234 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643238 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643240 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643243 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643246 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 12:30:33.649478 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643248 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643251 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643254 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643256 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643259 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643262 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643265 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643268 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643270 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643273 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643276 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643278 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643281 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643283 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643285 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643288 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643290 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643292 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643295 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 12:30:33.649972 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643297 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643299 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643302 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643309 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643312 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643314 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643317 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:33.643319 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.643323 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.643434 2562 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.645405 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.646276 2562 server.go:1019] "Starting client certificate rotation" Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.646375 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 12:30:33.650691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.646411 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 12:30:33.651197 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.648542 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 12:30:33.653113 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.653093 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 12:30:33.661856 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.661840 2562 log.go:25] "Validated CRI v1 runtime API" Sep 10 12:30:33.664943 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.664928 2562 log.go:25] "Validated CRI v1 image API" Sep 10 12:30:33.665768 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.665742 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 12:30:33.669997 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.669973 2562 fs.go:135] Filesystem UUIDs: map[09248f55-a778-475b-9368-a06151847d85:/dev/nvme0n1p4 1a968618-51b7-4f4c-a64c-195875a07def:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 10 12:30:33.670048 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.669998 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 12:30:33.674082 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.673985 2562 manager.go:216] Machine: {Timestamp:2026-09-10 12:30:33.673514942 +0000 UTC m=+0.299712508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089562 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd016124b1a90a0bcc4af81fc88c0 SystemUUID:ec2bd016-124b-1a90-a0bc-c4af81fc88c0 BootID:802d27c7-d8b2-4376-b8c5-eb5bf46d6cce Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:71:b9:67:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:71:b9:67:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:d7:ad:f2:97:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 12:30:33.674082 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.674081 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 12:30:33.674196 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.674147 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 12:30:33.676823 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.676794 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 12:30:33.676948 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.676828 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 12:30:33.677991 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.677981 2562 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 12:30:33.678066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.677994 2562 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 12:30:33.678066 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.678003 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 12:30:33.678527 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.678512 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 12:30:33.678527 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.678523 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 12:30:33.679643 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.679633 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 12:30:33.683277 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683265 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 10 12:30:33.683395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683385 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 12:30:33.683438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683415 2562 kubelet.go:486] "Attempting to sync node with API server" Sep 10 12:30:33.683438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683429 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 12:30:33.683492 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683441 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 12:30:33.683492 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683450 2562 kubelet.go:398] "Adding apiserver pod source" Sep 10 12:30:33.683492 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683458 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 12:30:33.683825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683813 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 12:30:33.683868 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.683834 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 12:30:33.686637 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.686614 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 12:30:33.689181 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689060 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 12:30:33.689260 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689198 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 12:30:33.689353 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689343 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 12:30:33.689384 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689357 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 12:30:33.689384 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689363 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 12:30:33.689384 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689368 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 12:30:33.689384 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689376 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 12:30:33.689384 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689381 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 12:30:33.689813 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689804 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 12:30:33.689813 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689813 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 12:30:33.689884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689821 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 12:30:33.689884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689827 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 12:30:33.689884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689835 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 12:30:33.689884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689842 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 12:30:33.689884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689864 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 12:30:33.689884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.689870 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 12:30:33.692942 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.692916 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 12:30:33.693033 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.692990 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 12:30:33.693088 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.693045 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 12:30:33.693542 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.693529 2562 server.go:1268] "Started kubelet" Sep 10 12:30:33.693655 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.693599 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 12:30:33.693716 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.693637 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 12:30:33.693716 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.693693 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 12:30:33.694289 ip-10-0-137-89 systemd[1]: Started Kubernetes Kubelet. Sep 10 12:30:33.695243 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.695218 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 12:30:33.697718 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.697696 2562 server.go:310] "Adding debug handlers to kubelet server" Sep 10 12:30:33.700083 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.700066 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 12:30:33.700222 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.700132 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 12:30:33.700626 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.700612 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 12:30:33.701124 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.700776 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:33.701362 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.700467 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-89.ec2.internal.18d3f586310bd5a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-89.ec2.internal,UID:ip-10-0-137-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-89.ec2.internal,},FirstTimestamp:2026-09-10 12:30:33.693500835 +0000 UTC m=+0.319698401,LastTimestamp:2026-09-10 12:30:33.693500835 +0000 UTC m=+0.319698401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-89.ec2.internal,}" Sep 10 12:30:33.701458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701397 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 12:30:33.701458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701397 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 12:30:33.701458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701417 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 12:30:33.701608 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701596 2562 reconstruct.go:97] "Volume reconstruction finished" Sep 10 12:30:33.701662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701608 2562 reconciler.go:29] "Reconciler: start to sync state" Sep 10 12:30:33.701662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701648 2562 factory.go:55] Registering systemd factory Sep 10 12:30:33.701662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701663 2562 factory.go:223] Registration of the systemd container factory successfully Sep 10 12:30:33.701884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701866 2562 factory.go:153] Registering CRI-O factory Sep 10 12:30:33.701884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701883 2562 factory.go:223] Registration of the crio container factory successfully Sep 10 12:30:33.702026 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701931 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 12:30:33.702026 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701944 2562 factory.go:103] Registering Raw factory Sep 10 12:30:33.702026 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.701953 2562 manager.go:1196] Started watching for new ooms in manager Sep 10 12:30:33.702689 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.702674 2562 manager.go:349] Starting recovery of all containers Sep 10 12:30:33.711819 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.711787 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 12:30:33.711980 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.711956 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 12:30:33.714084 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.714067 2562 manager.go:354] Recovery completed Sep 10 12:30:33.715613 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.715537 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ctzpw" Sep 10 12:30:33.716091 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.716066 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 10 12:30:33.719059 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.719043 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.721410 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721391 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.721482 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721424 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.721482 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721437 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.721950 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721932 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 12:30:33.721950 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721949 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 12:30:33.722050 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721965 2562 state_mem.go:36] "Initialized new in-memory state store" Sep 10 12:30:33.722050 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.721964 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.722761 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.722744 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.722819 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.722778 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.722819 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.722791 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.723342 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.723281 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-89.ec2.internal.18d3f58632b5b0e1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-89.ec2.internal,UID:ip-10-0-137-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-89.ec2.internal,},FirstTimestamp:2026-09-10 12:30:33.721409761 +0000 UTC m=+0.347607330,LastTimestamp:2026-09-10 12:30:33.721409761 +0000 UTC m=+0.347607330,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-89.ec2.internal,}" Sep 10 12:30:33.723727 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.723714 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ctzpw" Sep 10 12:30:33.724247 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.724236 2562 policy_none.go:49] "None policy: Start" Sep 10 12:30:33.724276 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.724253 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 12:30:33.724276 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.724262 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 12:30:33.726517 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.726504 2562 policy_none.go:47] "Start" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769120 2562 manager.go:337] "Starting Device Plugin manager" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.769153 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769166 2562 server.go:87] "Starting device plugin registration server" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769451 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769463 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769575 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769667 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.769676 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.770103 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 12:30:33.780558 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.770169 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:33.830996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.830969 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 12:30:33.832159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.832147 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 12:30:33.832222 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.832194 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 12:30:33.832222 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.832214 2562 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 12:30:33.832281 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.832246 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 12:30:33.834692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.834671 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 12:30:33.869564 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.869538 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.870397 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.870380 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.870469 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.870416 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.870469 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.870426 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.870469 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.870448 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-89.ec2.internal" Sep 10 12:30:33.885514 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.885433 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-89.ec2.internal" Sep 10 12:30:33.885514 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.885458 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-89.ec2.internal\": node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:33.916716 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.916693 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:33.932778 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.932750 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal"] Sep 10 12:30:33.932881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.932846 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.933748 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.933734 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.933814 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.933759 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.933814 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.933770 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.935851 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.935839 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.935959 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.935945 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:33.936003 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.935973 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.936575 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.936559 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.936641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.936559 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.936641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.936612 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.936641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.936621 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.936641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.936624 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.936641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.936635 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.938632 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.938617 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" Sep 10 12:30:33.938686 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.938646 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 12:30:33.939309 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.939293 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientMemory" Sep 10 12:30:33.939371 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.939321 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 12:30:33.939371 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:33.939335 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeHasSufficientPID" Sep 10 12:30:33.953120 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.953102 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-89.ec2.internal\" not found" node="ip-10-0-137-89.ec2.internal" Sep 10 12:30:33.957364 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:33.957345 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-89.ec2.internal\" not found" node="ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.002913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.002890 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8daec91b6e7f56892ca7ae4f5c0cc8e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal\" (UID: \"8daec91b6e7f56892ca7ae4f5c0cc8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.003005 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.002916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8daec91b6e7f56892ca7ae4f5c0cc8e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal\" (UID: \"8daec91b6e7f56892ca7ae4f5c0cc8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.003005 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.002932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05dc69e2d25cad8ef7303fdcd6d064db-config\") pod \"kube-apiserver-proxy-ip-10-0-137-89.ec2.internal\" (UID: \"05dc69e2d25cad8ef7303fdcd6d064db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.017228 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.017212 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.103554 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.103535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8daec91b6e7f56892ca7ae4f5c0cc8e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal\" (UID: \"8daec91b6e7f56892ca7ae4f5c0cc8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.103638 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.103557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8daec91b6e7f56892ca7ae4f5c0cc8e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal\" (UID: \"8daec91b6e7f56892ca7ae4f5c0cc8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.103638 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.103574 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05dc69e2d25cad8ef7303fdcd6d064db-config\") pod \"kube-apiserver-proxy-ip-10-0-137-89.ec2.internal\" (UID: \"05dc69e2d25cad8ef7303fdcd6d064db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.103638 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.103629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8daec91b6e7f56892ca7ae4f5c0cc8e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal\" (UID: \"8daec91b6e7f56892ca7ae4f5c0cc8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.103735 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.103683 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8daec91b6e7f56892ca7ae4f5c0cc8e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal\" (UID: \"8daec91b6e7f56892ca7ae4f5c0cc8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.103735 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.103706 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/05dc69e2d25cad8ef7303fdcd6d064db-config\") pod \"kube-apiserver-proxy-ip-10-0-137-89.ec2.internal\" (UID: \"05dc69e2d25cad8ef7303fdcd6d064db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.117655 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.117638 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.218478 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.218431 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.255612 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.255572 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.260824 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.260495 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" Sep 10 12:30:34.319470 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.319451 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.419914 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.419888 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.520499 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.520451 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.620911 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.620889 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.646060 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.646042 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 12:30:34.646413 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.646177 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 12:30:34.717338 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.717310 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 12:30:34.721140 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.721121 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.726273 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.726253 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 12:25:34 +0000 UTC" deadline="2028-03-05 05:14:11.460134651 +0000 UTC" Sep 10 12:30:34.726323 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.726274 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13000h43m36.733864445s" Sep 10 12:30:34.821957 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.821929 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.922528 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:34.922495 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:34.951070 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:34.951044 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8daec91b6e7f56892ca7ae4f5c0cc8e1.slice/crio-44a697effe2fddcda2bea98d84424e426c2ee4a558ea4069c5d11baf2e2b8eb7 WatchSource:0}: Error finding container 44a697effe2fddcda2bea98d84424e426c2ee4a558ea4069c5d11baf2e2b8eb7: Status 404 returned error can't find the container with id 44a697effe2fddcda2bea98d84424e426c2ee4a558ea4069c5d11baf2e2b8eb7 Sep 10 12:30:34.956814 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:34.956791 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:30:34.987563 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:34.987541 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dc69e2d25cad8ef7303fdcd6d064db.slice/crio-bef6f1ec9a02edeef388a4408266ac61c527ebf36ff805e829aa1ea849f78fbc WatchSource:0}: Error finding container bef6f1ec9a02edeef388a4408266ac61c527ebf36ff805e829aa1ea849f78fbc: Status 404 returned error can't find the container with id bef6f1ec9a02edeef388a4408266ac61c527ebf36ff805e829aa1ea849f78fbc Sep 10 12:30:35.022955 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.022932 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:35.123532 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.123479 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-89.ec2.internal\" not found" Sep 10 12:30:35.137083 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.137063 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 12:30:35.195983 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.195959 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 12:30:35.201358 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.201340 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" Sep 10 12:30:35.216658 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.216640 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 12:30:35.217607 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.217593 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" Sep 10 12:30:35.225177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.225160 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 12:30:35.684663 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.684463 2562 apiserver.go:52] "Watching apiserver" Sep 10 12:30:35.692956 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.692921 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 12:30:35.693316 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.693292 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6s69q","openshift-network-diagnostics/network-check-target-twpc8","openshift-network-operator/iptables-alerter-7qq24","openshift-ovn-kubernetes/ovnkube-node-t5558","kube-system/global-pull-secret-syncer-5s256","kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5","openshift-cluster-node-tuning-operator/tuned-xl7bp","openshift-dns/node-resolver-5kcqc","openshift-image-registry/node-ca-zz8th","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal","openshift-multus/multus-jhrk2","kube-system/konnectivity-agent-mmnx4","openshift-multus/multus-additional-cni-plugins-vkghf"] Sep 10 12:30:35.696063 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.696027 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.698470 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.698450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.698984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.698958 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 12:30:35.698984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.698977 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.699282 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.699256 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.699349 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.699320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j7vwc\"" Sep 10 12:30:35.700812 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.700793 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 12:30:35.701635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.701616 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 12:30:35.701835 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.701819 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 12:30:35.702255 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.702234 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-khq45\"" Sep 10 12:30:35.702459 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.702438 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.703423 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.703404 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.705061 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.705040 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.705181 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.705156 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 12:30:35.705309 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.705182 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fhkcw\"" Sep 10 12:30:35.705467 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.705445 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.705881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.705789 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.705881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.705812 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 12:30:35.706104 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.706085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.706264 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.706235 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:35.706264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.706263 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.706600 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.706570 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 12:30:35.706727 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.706579 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 12:30:35.706727 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.706610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztcjf\"" Sep 10 12:30:35.707919 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.707900 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 12:30:35.708333 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.708306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.711272 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.711370 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.711394 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712071 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712086 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712128 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.712780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712377 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2s8w\"" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712870 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-systemd\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-var-lib-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.712969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-socket-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-device-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-ovnkube-config\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-ovnkube-script-lib\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0fcf238-0942-47a0-b348-896e5049a982-konnectivity-ca\") pod \"konnectivity-agent-mmnx4\" (UID: \"f0fcf238-0942-47a0-b348-896e5049a982\") " pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.713178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-kubelet\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-systemd-units\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-node-log\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbcl\" (UniqueName: \"kubernetes.io/projected/17566da3-e5fb-45ca-891a-e1ec4796759a-kube-api-access-msbcl\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-registration-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0fcf238-0942-47a0-b348-896e5049a982-agent-certs\") pod \"konnectivity-agent-mmnx4\" (UID: \"f0fcf238-0942-47a0-b348-896e5049a982\") " pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7bx\" (UniqueName: \"kubernetes.io/projected/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-kube-api-access-sl7bx\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-run-netns\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-log-socket\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-host-slash\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.713635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713624 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-sys-fs\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-iptables-alerter-script\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-cni-netd\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17566da3-e5fb-45ca-891a-e1ec4796759a-ovn-node-metrics-cert\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2rk\" (UniqueName: \"kubernetes.io/projected/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-kube-api-access-zx2rk\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713877 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-slash\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-run-ovn-kubernetes\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-cni-bin\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.713981 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-env-overrides\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.714074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-etc-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.714112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.714144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-ovn\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714198 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.714184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.714940 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.714680 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:35.714940 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.714778 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:35.715241 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.715204 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 12:30:35.715487 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.715467 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.715666 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.715534 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lgtl6\"" Sep 10 12:30:35.715743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.715727 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.717631 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.717428 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:35.717631 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.717498 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:35.720019 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.719971 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.722561 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.722539 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.722971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.722769 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.722971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.722796 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f2rxn\"" Sep 10 12:30:35.722971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.722795 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.722971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.722795 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 12:30:35.725260 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.725242 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.725709 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.725693 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 12:30:35.725913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.725898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pg7jb\"" Sep 10 12:30:35.726009 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.725989 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 12:30:35.728034 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.728016 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 12:30:35.728130 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.728114 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hgct5\"" Sep 10 12:30:35.741721 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.741704 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhjxr" Sep 10 12:30:35.753016 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.752993 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhjxr" Sep 10 12:30:35.802718 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.802687 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 12:30:35.815060 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-kubelet\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.815212 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-run\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.815212 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815155 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-daemon-config\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.815212 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815164 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-kubelet\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.815375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815234 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-registration-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.815375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7bx\" (UniqueName: \"kubernetes.io/projected/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-kube-api-access-sl7bx\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.815375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-multus-certs\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.815504 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-registration-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.815504 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-run-netns\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.815504 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-sys-fs\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.815504 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-iptables-alerter-script\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-cni-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815521 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-run-netns\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815537 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-conf-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-cnibin\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-sys-fs\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-cni-binary-copy\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17566da3-e5fb-45ca-891a-e1ec4796759a-ovn-node-metrics-cert\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.815737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815737 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46be07f1-b76e-451e-a75e-0ac2af42c40a-kubelet-config\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysctl-conf\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-system-cni-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-etc-kubernetes\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-slash\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-slash\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.815989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57937ad4-a410-4961-8ba1-e72addf6c5a3-hosts-file\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816009 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wx87\" (UniqueName: \"kubernetes.io/projected/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-kube-api-access-7wx87\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-host\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.816073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-etc-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-etc-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysctl-d\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816173 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-sys\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/397fbfb0-a6e3-4df2-9be2-b08d939993b4-cni-binary-copy\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pgln\" (UniqueName: \"kubernetes.io/projected/397fbfb0-a6e3-4df2-9be2-b08d939993b4-kube-api-access-9pgln\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816284 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-kubernetes\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-systemd\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816344 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-var-lib-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816365 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-systemd\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-socket-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.816617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-device-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-var-lib-openvswitch\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9dw\" (UniqueName: \"kubernetes.io/projected/57937ad4-a410-4961-8ba1-e72addf6c5a3-kube-api-access-xg9dw\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-socket-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816507 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-cnibin\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-device-dir\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46be07f1-b76e-451e-a75e-0ac2af42c40a-dbus\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-lib-modules\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-iptables-alerter-script\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.816564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-var-lib-kubelet\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-tuned\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-socket-dir-parent\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817112 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-systemd-units\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817260 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-node-log\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.817342 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbcl\" (UniqueName: \"kubernetes.io/projected/17566da3-e5fb-45ca-891a-e1ec4796759a-kube-api-access-msbcl\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysconfig\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817332 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-systemd-units\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289cv\" (UniqueName: \"kubernetes.io/projected/fd98d133-19c5-416b-ba3d-ef46bf070e48-kube-api-access-289cv\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0fcf238-0942-47a0-b348-896e5049a982-agent-certs\") pod \"konnectivity-agent-mmnx4\" (UID: \"f0fcf238-0942-47a0-b348-896e5049a982\") " pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-cni-multus\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-log-socket\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-host-slash\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-cni-netd\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817599 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-modprobe-d\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817669 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-node-log\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-k8s-cni-cncf-io\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2rk\" (UniqueName: \"kubernetes.io/projected/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-kube-api-access-zx2rk\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817707 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-log-socket\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-run-ovn-kubernetes\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817779 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-host-slash\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817852 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-cni-netd\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-cni-bin\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-env-overrides\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.817974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57937ad4-a410-4961-8ba1-e72addf6c5a3-tmp-dir\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818019 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-cni-bin\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-env-overrides\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-ovn\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-host-run-ovn-kubernetes\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-systemd\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818544 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-hostroot\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818584 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17566da3-e5fb-45ca-891a-e1ec4796759a-run-ovn\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-system-cni-dir\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818636 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226e7671-34bf-4221-9490-509ff6a6ab6e-tmp\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-ovnkube-config\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-ovnkube-script-lib\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.818984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-netns\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818860 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-cni-bin\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818896 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-kubelet\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-host\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818945 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-serviceca\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.818971 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0fcf238-0942-47a0-b348-896e5049a982-konnectivity-ca\") pod \"konnectivity-agent-mmnx4\" (UID: \"f0fcf238-0942-47a0-b348-896e5049a982\") " pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.819015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjsl\" (UniqueName: \"kubernetes.io/projected/226e7671-34bf-4221-9490-509ff6a6ab6e-kube-api-access-wtjsl\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.819042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbr7\" (UniqueName: \"kubernetes.io/projected/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-kube-api-access-2tbr7\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.819487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-ovnkube-script-lib\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.819672 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f0fcf238-0942-47a0-b348-896e5049a982-konnectivity-ca\") pod \"konnectivity-agent-mmnx4\" (UID: \"f0fcf238-0942-47a0-b348-896e5049a982\") " pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.819846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.819784 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17566da3-e5fb-45ca-891a-e1ec4796759a-ovnkube-config\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.821470 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.821443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17566da3-e5fb-45ca-891a-e1ec4796759a-ovn-node-metrics-cert\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.823147 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.822799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f0fcf238-0942-47a0-b348-896e5049a982-agent-certs\") pod \"konnectivity-agent-mmnx4\" (UID: \"f0fcf238-0942-47a0-b348-896e5049a982\") " pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:35.835282 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.835198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbcl\" (UniqueName: \"kubernetes.io/projected/17566da3-e5fb-45ca-891a-e1ec4796759a-kube-api-access-msbcl\") pod \"ovnkube-node-t5558\" (UID: \"17566da3-e5fb-45ca-891a-e1ec4796759a\") " pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:35.835390 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.835317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7bx\" (UniqueName: \"kubernetes.io/projected/ba58ecfe-97d8-4d2a-a596-33d5e5c80752-kube-api-access-sl7bx\") pod \"iptables-alerter-7qq24\" (UID: \"ba58ecfe-97d8-4d2a-a596-33d5e5c80752\") " pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:35.839160 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.839084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2rk\" (UniqueName: \"kubernetes.io/projected/4a8f6a5b-6e90-4209-bcca-906f47ab34c0-kube-api-access-zx2rk\") pod \"aws-ebs-csi-driver-node-vdww5\" (UID: \"4a8f6a5b-6e90-4209-bcca-906f47ab34c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:35.843978 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.841769 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" event={"ID":"05dc69e2d25cad8ef7303fdcd6d064db","Type":"ContainerStarted","Data":"bef6f1ec9a02edeef388a4408266ac61c527ebf36ff805e829aa1ea849f78fbc"} Sep 10 12:30:35.844461 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.844437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" event={"ID":"8daec91b6e7f56892ca7ae4f5c0cc8e1","Type":"ContainerStarted","Data":"44a697effe2fddcda2bea98d84424e426c2ee4a558ea4069c5d11baf2e2b8eb7"} Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226e7671-34bf-4221-9490-509ff6a6ab6e-tmp\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-netns\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-cni-bin\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-kubelet\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-host\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-serviceca\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjsl\" (UniqueName: \"kubernetes.io/projected/226e7671-34bf-4221-9490-509ff6a6ab6e-kube-api-access-wtjsl\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbr7\" (UniqueName: \"kubernetes.io/projected/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-kube-api-access-2tbr7\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-run\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.919989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-daemon-config\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-multus-certs\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-cni-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-conf-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-cnibin\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-cni-binary-copy\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.920866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46be07f1-b76e-451e-a75e-0ac2af42c40a-kubelet-config\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysctl-conf\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-system-cni-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-etc-kubernetes\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57937ad4-a410-4961-8ba1-e72addf6c5a3-hosts-file\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wx87\" (UniqueName: \"kubernetes.io/projected/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-kube-api-access-7wx87\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-host\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920354 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysctl-d\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-sys\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/397fbfb0-a6e3-4df2-9be2-b08d939993b4-cni-binary-copy\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pgln\" (UniqueName: \"kubernetes.io/projected/397fbfb0-a6e3-4df2-9be2-b08d939993b4-kube-api-access-9pgln\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920457 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-kubernetes\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9dw\" (UniqueName: \"kubernetes.io/projected/57937ad4-a410-4961-8ba1-e72addf6c5a3-kube-api-access-xg9dw\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-cnibin\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46be07f1-b76e-451e-a75e-0ac2af42c40a-dbus\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-lib-modules\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.921889 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-var-lib-kubelet\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-tuned\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920717 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-socket-dir-parent\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysconfig\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920830 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289cv\" (UniqueName: \"kubernetes.io/projected/fd98d133-19c5-416b-ba3d-ef46bf070e48-kube-api-access-289cv\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-cni-multus\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-modprobe-d\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-k8s-cni-cncf-io\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.920992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57937ad4-a410-4961-8ba1-e72addf6c5a3-tmp-dir\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-systemd\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921056 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-hostroot\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-system-cni-dir\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-netns\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-run\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.922826 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921789 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-daemon-config\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-multus-certs\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.921968 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-cni-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922001 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-conf-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-cni-binary-copy\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/46be07f1-b76e-451e-a75e-0ac2af42c40a-kubelet-config\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922082 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-multus-socket-dir-parent\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922108 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-etc-kubernetes\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-system-cni-dir\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57937ad4-a410-4961-8ba1-e72addf6c5a3-hosts-file\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.922210 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922221 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysctl-conf\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.922514 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:30:36.422474219 +0000 UTC m=+3.048671797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-lib-modules\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-cnibin\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/46be07f1-b76e-451e-a75e-0ac2af42c40a-dbus\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:35.923692 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-system-cni-dir\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.922980 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-cni-bin\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-kubelet\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-host\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fd98d133-19c5-416b-ba3d-ef46bf070e48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923414 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysconfig\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923461 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-kubernetes\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-serviceca\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.923583 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.923633 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:36.423617055 +0000 UTC m=+3.049814611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-var-lib-cni-multus\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923771 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-modprobe-d\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-host\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923924 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fd98d133-19c5-416b-ba3d-ef46bf070e48-cnibin\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923931 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-host-run-k8s-cni-cncf-io\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.923988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-sys\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.924436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.924268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-sysctl-d\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.925280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.924276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/397fbfb0-a6e3-4df2-9be2-b08d939993b4-cni-binary-copy\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.925280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.924442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/397fbfb0-a6e3-4df2-9be2-b08d939993b4-hostroot\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:35.925280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.924513 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-systemd\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.925280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.924572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57937ad4-a410-4961-8ba1-e72addf6c5a3-tmp-dir\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.925280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.924646 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/226e7671-34bf-4221-9490-509ff6a6ab6e-var-lib-kubelet\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.927409 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.927382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/226e7671-34bf-4221-9490-509ff6a6ab6e-etc-tuned\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.927917 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.927893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/226e7671-34bf-4221-9490-509ff6a6ab6e-tmp\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.929497 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.929470 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:30:35.929603 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.929503 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:30:35.929603 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.929518 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:35.929603 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:35.929586 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:36.429567205 +0000 UTC m=+3.055764777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:35.932435 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.932355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9dw\" (UniqueName: \"kubernetes.io/projected/57937ad4-a410-4961-8ba1-e72addf6c5a3-kube-api-access-xg9dw\") pod \"node-resolver-5kcqc\" (UID: \"57937ad4-a410-4961-8ba1-e72addf6c5a3\") " pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:35.934858 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.934831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbr7\" (UniqueName: \"kubernetes.io/projected/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-kube-api-access-2tbr7\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:35.935976 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.935926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wx87\" (UniqueName: \"kubernetes.io/projected/9a6dfcda-b4c5-41c0-9c31-53995fb29b8c-kube-api-access-7wx87\") pod \"node-ca-zz8th\" (UID: \"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c\") " pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:35.936763 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.936734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289cv\" (UniqueName: \"kubernetes.io/projected/fd98d133-19c5-416b-ba3d-ef46bf070e48-kube-api-access-289cv\") pod \"multus-additional-cni-plugins-vkghf\" (UID: \"fd98d133-19c5-416b-ba3d-ef46bf070e48\") " pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:35.938906 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.938885 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjsl\" (UniqueName: \"kubernetes.io/projected/226e7671-34bf-4221-9490-509ff6a6ab6e-kube-api-access-wtjsl\") pod \"tuned-xl7bp\" (UID: \"226e7671-34bf-4221-9490-509ff6a6ab6e\") " pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:35.939062 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:35.939039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pgln\" (UniqueName: \"kubernetes.io/projected/397fbfb0-a6e3-4df2-9be2-b08d939993b4-kube-api-access-9pgln\") pod \"multus-jhrk2\" (UID: \"397fbfb0-a6e3-4df2-9be2-b08d939993b4\") " pod="openshift-multus/multus-jhrk2" Sep 10 12:30:36.010260 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.010225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" Sep 10 12:30:36.020088 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.020060 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:36.030948 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.030925 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7qq24" Sep 10 12:30:36.037099 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.037081 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:30:36.044243 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.044079 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vkghf" Sep 10 12:30:36.051913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.051890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5kcqc" Sep 10 12:30:36.059059 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.059019 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zz8th" Sep 10 12:30:36.066011 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.065987 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" Sep 10 12:30:36.071020 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.071003 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jhrk2" Sep 10 12:30:36.205331 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.205306 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 12:30:36.425633 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.425557 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:36.425818 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.425644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:36.425818 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.425747 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:36.425818 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.425751 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:36.425818 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.425802 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:37.425786636 +0000 UTC m=+4.051984190 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:36.426042 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.425822 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:30:37.42581287 +0000 UTC m=+4.052010425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:36.526142 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.526108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:36.526329 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.526293 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:30:36.526329 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.526313 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:30:36.526329 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.526324 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:36.526486 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:36.526383 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:37.526367998 +0000 UTC m=+4.152565553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:36.758145 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.758051 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:25:36 +0000 UTC" deadline="2028-03-10 02:35:14.108609293 +0000 UTC" Sep 10 12:30:36.758145 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:36.758078 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13118h4m37.350534824s" Sep 10 12:30:37.091302 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:37.091272 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57937ad4_a410_4961_8ba1_e72addf6c5a3.slice/crio-96b9c2b8b66637b368a8b5bb07a2b4cfbec43e0eb2f70814ef6e6d995f8884f0 WatchSource:0}: Error finding container 96b9c2b8b66637b368a8b5bb07a2b4cfbec43e0eb2f70814ef6e6d995f8884f0: Status 404 returned error can't find the container with id 96b9c2b8b66637b368a8b5bb07a2b4cfbec43e0eb2f70814ef6e6d995f8884f0 Sep 10 12:30:37.092599 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:37.092387 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397fbfb0_a6e3_4df2_9be2_b08d939993b4.slice/crio-47b34d92353bee77caa119dd2150b721b66d3deb4eb2ffffb9e59896757bd4e2 WatchSource:0}: Error finding container 47b34d92353bee77caa119dd2150b721b66d3deb4eb2ffffb9e59896757bd4e2: Status 404 returned error can't find the container with id 47b34d92353bee77caa119dd2150b721b66d3deb4eb2ffffb9e59896757bd4e2 Sep 10 12:30:37.093940 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:37.093908 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8f6a5b_6e90_4209_bcca_906f47ab34c0.slice/crio-5e8f8adf90bbe478e0c5bf3cfe7a359b3e857e54f443e7cc7e7f2810a42ea56a WatchSource:0}: Error finding container 5e8f8adf90bbe478e0c5bf3cfe7a359b3e857e54f443e7cc7e7f2810a42ea56a: Status 404 returned error can't find the container with id 5e8f8adf90bbe478e0c5bf3cfe7a359b3e857e54f443e7cc7e7f2810a42ea56a Sep 10 12:30:37.114232 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:37.110162 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17566da3_e5fb_45ca_891a_e1ec4796759a.slice/crio-60f180457fb5221ed283d2790448e042509dd6098696117f7a0605fc6208a9aa WatchSource:0}: Error finding container 60f180457fb5221ed283d2790448e042509dd6098696117f7a0605fc6208a9aa: Status 404 returned error can't find the container with id 60f180457fb5221ed283d2790448e042509dd6098696117f7a0605fc6208a9aa Sep 10 12:30:37.115541 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:30:37.115012 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba58ecfe_97d8_4d2a_a596_33d5e5c80752.slice/crio-f8d8a55c0969d39784540c16069ea49b47bee94e2d4c814c04f225f34670affc WatchSource:0}: Error finding container f8d8a55c0969d39784540c16069ea49b47bee94e2d4c814c04f225f34670affc: Status 404 returned error can't find the container with id f8d8a55c0969d39784540c16069ea49b47bee94e2d4c814c04f225f34670affc Sep 10 12:30:37.434332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.434302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:37.434449 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.434363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:37.434492 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.434448 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:37.434492 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.434447 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:37.434557 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.434496 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:39.434482244 +0000 UTC m=+6.060679799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:37.434557 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.434510 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:30:39.434503153 +0000 UTC m=+6.060700708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:37.534700 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.534668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:37.534835 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.534804 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:30:37.534835 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.534824 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:30:37.534968 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.534871 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:37.534968 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.534955 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:39.534909684 +0000 UTC m=+6.161107254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:37.761851 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.761740 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:25:36 +0000 UTC" deadline="2028-05-28 16:09:43.321502452 +0000 UTC" Sep 10 12:30:37.761851 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.761806 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15027h39m5.559701013s" Sep 10 12:30:37.833808 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.833777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:37.833978 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.833812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:37.833978 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.833916 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:37.833978 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.833970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:37.834157 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.834072 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:37.834227 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:37.834182 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:37.857547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.857515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qq24" event={"ID":"ba58ecfe-97d8-4d2a-a596-33d5e5c80752","Type":"ContainerStarted","Data":"f8d8a55c0969d39784540c16069ea49b47bee94e2d4c814c04f225f34670affc"} Sep 10 12:30:37.863534 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.863502 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mmnx4" event={"ID":"f0fcf238-0942-47a0-b348-896e5049a982","Type":"ContainerStarted","Data":"911955a916742fdd3081071fce7c881e10d0d0b29f44f37aaf40f0c9fcf5f649"} Sep 10 12:30:37.871935 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.871636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" event={"ID":"226e7671-34bf-4221-9490-509ff6a6ab6e","Type":"ContainerStarted","Data":"178d47257076801bbdf9cf841a228b82d37c75c057a89bdc18771b74e7e8e009"} Sep 10 12:30:37.874251 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.874205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zz8th" event={"ID":"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c","Type":"ContainerStarted","Data":"6869a8642501aaa8ebca4735e43f610a983fe73753de08a83033ce8004515981"} Sep 10 12:30:37.876359 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.876258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerStarted","Data":"03859843d7599ae94259df4babf1283e3a88c9615d84e53f670bda896fa75b51"} Sep 10 12:30:37.879284 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.879156 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jhrk2" event={"ID":"397fbfb0-a6e3-4df2-9be2-b08d939993b4","Type":"ContainerStarted","Data":"47b34d92353bee77caa119dd2150b721b66d3deb4eb2ffffb9e59896757bd4e2"} Sep 10 12:30:37.884630 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.884585 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" event={"ID":"05dc69e2d25cad8ef7303fdcd6d064db","Type":"ContainerStarted","Data":"b9e65a3757a6f5fdff9724a57914fa7149915761062293c50a9ec91fc56e454c"} Sep 10 12:30:37.893735 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.893709 2562 generic.go:356] "Generic (PLEG): container finished" podID="8daec91b6e7f56892ca7ae4f5c0cc8e1" containerID="e3b48d316721a6012a5905d2768ea672f5ed921afaffdfa35a9ed527d1995068" exitCode=0 Sep 10 12:30:37.893845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.893798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" event={"ID":"8daec91b6e7f56892ca7ae4f5c0cc8e1","Type":"ContainerDied","Data":"e3b48d316721a6012a5905d2768ea672f5ed921afaffdfa35a9ed527d1995068"} Sep 10 12:30:37.902431 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.902323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"60f180457fb5221ed283d2790448e042509dd6098696117f7a0605fc6208a9aa"} Sep 10 12:30:37.908566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.908484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" event={"ID":"4a8f6a5b-6e90-4209-bcca-906f47ab34c0","Type":"ContainerStarted","Data":"5e8f8adf90bbe478e0c5bf3cfe7a359b3e857e54f443e7cc7e7f2810a42ea56a"} Sep 10 12:30:37.915733 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.915704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5kcqc" event={"ID":"57937ad4-a410-4961-8ba1-e72addf6c5a3","Type":"ContainerStarted","Data":"96b9c2b8b66637b368a8b5bb07a2b4cfbec43e0eb2f70814ef6e6d995f8884f0"} Sep 10 12:30:37.927882 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:37.926388 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-89.ec2.internal" podStartSLOduration=2.926368191 podStartE2EDuration="2.926368191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:37.904790134 +0000 UTC m=+4.530987712" watchObservedRunningTime="2026-09-10 12:30:37.926368191 +0000 UTC m=+4.552565770" Sep 10 12:30:38.947322 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:38.947284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" event={"ID":"8daec91b6e7f56892ca7ae4f5c0cc8e1","Type":"ContainerStarted","Data":"db9cabbdd27a2fb1d1bba501808376f865029676f7dadf3e09b8f3acd2cb238c"} Sep 10 12:30:39.451683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:39.451031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:39.451683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:39.451112 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:39.451683 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.451261 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:39.451683 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.451325 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:30:43.451305374 +0000 UTC m=+10.077502934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:39.452192 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.451759 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:39.452192 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.451810 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:43.451794325 +0000 UTC m=+10.077991884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:39.552388 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:39.552352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:39.552569 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.552506 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:30:39.552569 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.552529 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:30:39.552569 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.552541 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:39.552766 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.552599 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:43.552580491 +0000 UTC m=+10.178778061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:39.834361 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:39.833749 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:39.834361 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:39.833764 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:39.834361 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.833894 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:39.834361 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:39.833946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:39.834361 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.834056 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:39.834361 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:39.834141 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:41.833074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:41.833044 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:41.833523 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:41.833107 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:41.833523 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:41.833237 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:41.833523 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:41.833292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:41.833523 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:41.833363 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:41.834843 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:41.834787 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:43.484735 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:43.484694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:43.485213 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:43.484785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:43.485213 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.484828 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:43.485213 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.484919 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:43.485213 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.484925 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:30:51.484902872 +0000 UTC m=+18.111100443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:43.485213 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.484986 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:51.484969399 +0000 UTC m=+18.111166961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:43.585613 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:43.585576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:43.585845 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.585808 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:30:43.585845 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.585841 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:30:43.585983 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.585858 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:43.585983 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.585915 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:51.585895738 +0000 UTC m=+18.212093299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:43.834180 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:43.833322 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:43.834180 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.833467 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:43.834180 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:43.833896 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:43.834180 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.833981 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:43.834180 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:43.834034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:43.834180 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:43.834102 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:45.833404 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:45.833372 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:45.833862 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:45.833496 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:45.833940 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:45.833921 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:45.834043 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:45.834025 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:45.834109 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:45.834080 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:45.834164 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:45.834146 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:47.833718 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:47.833279 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:47.833718 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:47.833282 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:47.833718 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:47.833406 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:47.833718 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:47.833295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:47.833718 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:47.833471 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:47.833718 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:47.833613 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:49.832923 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:49.832874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:49.833362 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:49.832874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:49.833362 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:49.833012 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:49.833362 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:49.832874 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:49.833362 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:49.833072 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:49.833362 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:49.833152 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:51.547090 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:51.547061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:51.547633 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:51.547107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:51.547633 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.547222 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:51.547633 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.547234 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:51.547633 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.547285 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:31:07.547264959 +0000 UTC m=+34.173462512 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:30:51.547633 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.547299 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:07.547293634 +0000 UTC m=+34.173491186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:30:51.647874 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:51.647846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:51.648043 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.647986 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:30:51.648043 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.648008 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:30:51.648043 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.648020 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:51.648180 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.648079 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:07.648061236 +0000 UTC m=+34.274258809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:30:51.832862 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:51.832786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:51.832862 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:51.832818 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:51.833070 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:51.832786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:51.833070 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.832901 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:51.833070 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.833008 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:51.833208 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:51.833085 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:53.834116 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:53.834082 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:53.834559 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:53.834165 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:53.834559 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:53.834179 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:53.834559 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:53.834267 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:53.834559 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:53.834293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:53.834559 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:53.834361 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:55.839148 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.836885 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:55.839148 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:55.837005 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:55.839148 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.837079 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:55.839148 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:55.837161 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:55.839148 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.837219 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:55.839148 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:55.837339 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:55.984180 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.984009 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mmnx4" event={"ID":"f0fcf238-0942-47a0-b348-896e5049a982","Type":"ContainerStarted","Data":"78541a82794adca559143ecbebe4f61478ea866ba4f390345fe5492b4ce3e6fa"} Sep 10 12:30:55.985828 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.985799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" event={"ID":"226e7671-34bf-4221-9490-509ff6a6ab6e","Type":"ContainerStarted","Data":"665a355560004bbab84c1be5f58e98f765f89a9f84c6908ae17d3bc2d978a27b"} Sep 10 12:30:55.988383 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.988333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zz8th" event={"ID":"9a6dfcda-b4c5-41c0-9c31-53995fb29b8c","Type":"ContainerStarted","Data":"44d2d20ee5a7e65344c6b930969a8067af1ad7fed5c716090c525d8e253775cc"} Sep 10 12:30:55.990605 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.990560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerStarted","Data":"32aedbb789d1867d9fe983f32850185e6302a62b296c607893ab85331bea0749"} Sep 10 12:30:55.992164 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.992138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jhrk2" event={"ID":"397fbfb0-a6e3-4df2-9be2-b08d939993b4","Type":"ContainerStarted","Data":"9c396a14748204600278537d62119e3c7e965d479d1ce1a6939e349bdc916058"} Sep 10 12:30:55.994750 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.994712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"00b3161129b2931c261480e42e796dc6a81bf62772089aae8fed8b3d29078754"} Sep 10 12:30:55.994852 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.994784 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"fc2033a9e8bc1a53c0467d499a3a8c08db7704cb1d89c640d6b65e2b2cdcee21"} Sep 10 12:30:55.996500 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.996477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" event={"ID":"4a8f6a5b-6e90-4209-bcca-906f47ab34c0","Type":"ContainerStarted","Data":"eecb4524c1dd54e3b23c983bab746739a490a09e5e351b11cbd2d6ca380280bc"} Sep 10 12:30:55.998041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:55.997979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5kcqc" event={"ID":"57937ad4-a410-4961-8ba1-e72addf6c5a3","Type":"ContainerStarted","Data":"0260c1b54f0624c43aaf15bc01484a06d7b6d8ec73c9eb853200228f66e3796e"} Sep 10 12:30:56.000378 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:56.000342 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-89.ec2.internal" podStartSLOduration=21.000331506 podStartE2EDuration="21.000331506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:38.979013243 +0000 UTC m=+5.605210822" watchObservedRunningTime="2026-09-10 12:30:56.000331506 +0000 UTC m=+22.626529092" Sep 10 12:30:56.000538 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:56.000510 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mmnx4" podStartSLOduration=3.40247428 podStartE2EDuration="22.000502969s" totalImagesPullingTime="18.598028689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:56.000053934 +0000 UTC m=+22.626251529" watchObservedRunningTime="2026-09-10 12:30:56.000502969 +0000 UTC m=+22.626700547" Sep 10 12:30:56.017080 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:56.017033 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zz8th" podStartSLOduration=3.41528112 podStartE2EDuration="22.017020533s" totalImagesPullingTime="18.601739413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:56.016569511 +0000 UTC m=+22.642767090" watchObservedRunningTime="2026-09-10 12:30:56.017020533 +0000 UTC m=+22.643218126" Sep 10 12:30:56.056176 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:56.056106 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xl7bp" podStartSLOduration=3.456394676 podStartE2EDuration="22.056089242s" totalImagesPullingTime="18.599694566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:56.035072936 +0000 UTC m=+22.661270512" watchObservedRunningTime="2026-09-10 12:30:56.056089242 +0000 UTC m=+22.682286815" Sep 10 12:30:56.056335 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:56.056298 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jhrk2" podStartSLOduration=3.380991695 podStartE2EDuration="22.056291709s" totalImagesPullingTime="18.675300014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:56.055635361 +0000 UTC m=+22.681832937" watchObservedRunningTime="2026-09-10 12:30:56.056291709 +0000 UTC m=+22.682489283" Sep 10 12:30:57.002553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.002386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"766d40ea7d1b8d294f9a410dec421375ff2a251cb3a98ab0251b2de7b4b6cd85"} Sep 10 12:30:57.003207 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.002642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"be135b0504c106a730e7db0d09f027457da5b47c5871d380ffb91fc18790d3cc"} Sep 10 12:30:57.003207 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.002678 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"e3bc65797dd5efa48860ab7958c731d683e91a6e2416e758e20a343e751289e8"} Sep 10 12:30:57.003207 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.002712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"80189ca7d43203d5d51cb9eaa5214153541ce712c7fc9a4c21b0a7d5d917faea"} Sep 10 12:30:57.003847 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.003823 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7qq24" event={"ID":"ba58ecfe-97d8-4d2a-a596-33d5e5c80752","Type":"ContainerStarted","Data":"b394d38c90267edd326824d9f6ee11f3fcd517e9615c243fd83ddc489dbb74f3"} Sep 10 12:30:57.005262 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.005238 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd98d133-19c5-416b-ba3d-ef46bf070e48" containerID="32aedbb789d1867d9fe983f32850185e6302a62b296c607893ab85331bea0749" exitCode=0 Sep 10 12:30:57.005347 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.005305 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerDied","Data":"32aedbb789d1867d9fe983f32850185e6302a62b296c607893ab85331bea0749"} Sep 10 12:30:57.021966 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.021931 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5kcqc" podStartSLOduration=4.404628346 podStartE2EDuration="23.021917987s" totalImagesPullingTime="18.617289641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:56.100875947 +0000 UTC m=+22.727073523" watchObservedRunningTime="2026-09-10 12:30:57.021917987 +0000 UTC m=+23.648115561" Sep 10 12:30:57.022336 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.022307 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7qq24" podStartSLOduration=4.443671711 podStartE2EDuration="23.0222914s" totalImagesPullingTime="18.578619689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:57.02181589 +0000 UTC m=+23.648013465" watchObservedRunningTime="2026-09-10 12:30:57.0222914 +0000 UTC m=+23.648488975" Sep 10 12:30:57.272777 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.272755 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 12:30:57.781401 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.781319 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T12:30:57.272774814Z","UUID":"27e96509-9ddf-44e6-8650-1c6f0ce32907","Handler":null,"Name":"","Endpoint":""} Sep 10 12:30:57.782890 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.782869 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 12:30:57.783015 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.782901 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 12:30:57.832854 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.832827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:57.832854 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.832844 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:57.833054 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:57.832827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:57.833054 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:57.832945 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:57.833054 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:57.833009 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:57.833200 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:57.833069 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:30:58.010089 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:58.010053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" event={"ID":"4a8f6a5b-6e90-4209-bcca-906f47ab34c0","Type":"ContainerStarted","Data":"21b8347d4567fe0cfc8c81c4de4ca6fb4a62b4ce1419995b89428f843a4b583c"} Sep 10 12:30:58.132614 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:58.132514 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:58.133675 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:58.133649 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:59.016817 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.016774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"d063818a8c2503baddcf4e90ab4bcec370af88c9d4510768901037ab9c49817d"} Sep 10 12:30:59.019122 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.019083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" event={"ID":"4a8f6a5b-6e90-4209-bcca-906f47ab34c0","Type":"ContainerStarted","Data":"04748d811b58cec43cf423843b7bab6f97ed76987cdd8649090574af9c272681"} Sep 10 12:30:59.019405 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.019391 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:59.020375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.020349 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mmnx4" Sep 10 12:30:59.041038 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.041002 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vdww5" podStartSLOduration=4.283614749 podStartE2EDuration="25.040988467s" totalImagesPullingTime="20.757373718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:59.040968736 +0000 UTC m=+25.667166313" watchObservedRunningTime="2026-09-10 12:30:59.040988467 +0000 UTC m=+25.667186036" Sep 10 12:30:59.833014 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.832984 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:30:59.833174 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.832983 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:30:59.833174 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:59.833098 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:30:59.833269 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:59.833203 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:30:59.833269 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:30:59.832991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:30:59.833353 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:30:59.833299 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:31:01.028137 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:01.027900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" event={"ID":"17566da3-e5fb-45ca-891a-e1ec4796759a","Type":"ContainerStarted","Data":"27f647b08d832461631c88aa0ac18558300d08841c144de2bdc1ff3522091e58"} Sep 10 12:31:01.061463 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:01.061395 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" podStartSLOduration=8.430894082 podStartE2EDuration="27.061376114s" totalImagesPullingTime="18.630482032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:01.06092847 +0000 UTC m=+27.687126046" watchObservedRunningTime="2026-09-10 12:31:01.061376114 +0000 UTC m=+27.687573689" Sep 10 12:31:01.833076 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:01.832822 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:01.833076 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:01.832948 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:31:01.834553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:01.834170 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:01.834553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:01.834200 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:01.834553 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:01.834290 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:31:01.834553 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:01.834370 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:31:02.031518 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.031487 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:31:02.031982 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.031575 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:31:02.031982 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.031650 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:31:02.048835 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.048796 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:31:02.049245 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.049229 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:31:02.533990 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.533965 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5s256"] Sep 10 12:31:02.534105 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.534052 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:02.534161 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:02.534139 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:31:02.538166 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.538136 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twpc8"] Sep 10 12:31:02.538266 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.538205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:02.538332 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:02.538274 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:31:02.551197 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.551143 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6s69q"] Sep 10 12:31:02.551276 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:02.551217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:02.551318 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:02.551291 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:31:03.034627 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:03.034599 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd98d133-19c5-416b-ba3d-ef46bf070e48" containerID="892cdbde446cf70d0c6b2a813c0bcc47c9de10b82c8ff3c09b5ef573409d03cb" exitCode=0 Sep 10 12:31:03.035190 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:03.034677 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerDied","Data":"892cdbde446cf70d0c6b2a813c0bcc47c9de10b82c8ff3c09b5ef573409d03cb"} Sep 10 12:31:04.833312 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:04.833106 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:04.833702 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:04.833330 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:31:04.833702 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:04.833168 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:04.833702 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:04.833168 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:04.833702 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:04.833470 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:31:04.833702 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:04.833560 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:31:05.040237 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:05.040208 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd98d133-19c5-416b-ba3d-ef46bf070e48" containerID="5a8ae785be09b7e3a90a16504e45a2f61ae71cc395bc05eec3fa3e9624712a62" exitCode=0 Sep 10 12:31:05.040237 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:05.040247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerDied","Data":"5a8ae785be09b7e3a90a16504e45a2f61ae71cc395bc05eec3fa3e9624712a62"} Sep 10 12:31:06.832968 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:06.832934 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:06.833338 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:06.833036 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:06.833338 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:06.833041 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:31:06.833338 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:06.833066 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:06.833338 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:06.833137 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:31:06.833338 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:06.833242 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:31:07.046415 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:07.046389 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd98d133-19c5-416b-ba3d-ef46bf070e48" containerID="f148f77585bf329fdab3995a047f7f9dfe143a3f5fe8a729f168d8a794cc8feb" exitCode=0 Sep 10 12:31:07.046570 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:07.046433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerDied","Data":"f148f77585bf329fdab3995a047f7f9dfe143a3f5fe8a729f168d8a794cc8feb"} Sep 10 12:31:07.563970 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:07.563943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:07.564170 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:07.563988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:07.564170 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.564078 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 12:31:07.564170 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.564084 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:31:07.564170 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.564138 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret podName:46be07f1-b76e-451e-a75e-0ac2af42c40a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:39.564123457 +0000 UTC m=+66.190321025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret") pod "global-pull-secret-syncer-5s256" (UID: "46be07f1-b76e-451e-a75e-0ac2af42c40a") : object "kube-system"/"original-pull-secret" not registered Sep 10 12:31:07.564170 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.564152 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:31:39.564145495 +0000 UTC m=+66.190343048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 12:31:07.665162 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:07.665134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:07.665294 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.665247 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 12:31:07.665294 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.665261 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 12:31:07.665294 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.665270 2562 projected.go:196] Error preparing data for projected volume kube-api-access-9bpkg for pod openshift-network-diagnostics/network-check-target-twpc8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:31:07.665388 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:07.665310 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg podName:1110380b-db68-445a-9601-1f6dedd8c1c9 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:39.665298258 +0000 UTC m=+66.291495811 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bpkg" (UniqueName: "kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg") pod "network-check-target-twpc8" (UID: "1110380b-db68-445a-9601-1f6dedd8c1c9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 12:31:08.833160 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:08.833077 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:08.833160 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:08.833095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:08.833626 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:08.833210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:08.833626 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:08.833207 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6s69q" podUID="3c49d065-9a7d-4b66-b420-0bd16b7dfbae" Sep 10 12:31:08.833626 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:08.833305 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twpc8" podUID="1110380b-db68-445a-9601-1f6dedd8c1c9" Sep 10 12:31:08.833626 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:08.833464 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5s256" podUID="46be07f1-b76e-451e-a75e-0ac2af42c40a" Sep 10 12:31:09.586300 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.586235 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-89.ec2.internal" event="NodeReady" Sep 10 12:31:09.586449 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.586338 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 12:31:09.637312 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.637283 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jjtjh"] Sep 10 12:31:09.666549 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.666520 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-chwd4"] Sep 10 12:31:09.666681 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.666630 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.669178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.669147 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 12:31:09.669178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.669147 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 12:31:09.669388 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.669260 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c86jl\"" Sep 10 12:31:09.682595 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.682514 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjtjh"] Sep 10 12:31:09.682690 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.682639 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-chwd4"] Sep 10 12:31:09.682690 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.682573 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.686829 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.686810 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 12:31:09.686947 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.686858 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 12:31:09.687004 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.686968 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 12:31:09.687100 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.687085 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fv8rg\"" Sep 10 12:31:09.780155 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.780126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5f17aa-ec97-43cc-b367-e66e57ff9503-tmp-dir\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.780293 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.780196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbxg\" (UniqueName: \"kubernetes.io/projected/7c311b6e-3312-4db0-9623-7e4b662e08ca-kube-api-access-6mbxg\") pod \"ingress-canary-chwd4\" (UID: \"7c311b6e-3312-4db0-9623-7e4b662e08ca\") " pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.780293 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.780236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5f17aa-ec97-43cc-b367-e66e57ff9503-config-volume\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.780385 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.780304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l75rm\" (UniqueName: \"kubernetes.io/projected/7d5f17aa-ec97-43cc-b367-e66e57ff9503-kube-api-access-l75rm\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.780385 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.780340 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c311b6e-3312-4db0-9623-7e4b662e08ca-cert\") pod \"ingress-canary-chwd4\" (UID: \"7c311b6e-3312-4db0-9623-7e4b662e08ca\") " pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.780385 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.780370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.881531 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881450 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbxg\" (UniqueName: \"kubernetes.io/projected/7c311b6e-3312-4db0-9623-7e4b662e08ca-kube-api-access-6mbxg\") pod \"ingress-canary-chwd4\" (UID: \"7c311b6e-3312-4db0-9623-7e4b662e08ca\") " pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.881531 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5f17aa-ec97-43cc-b367-e66e57ff9503-config-volume\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.881531 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881516 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l75rm\" (UniqueName: \"kubernetes.io/projected/7d5f17aa-ec97-43cc-b367-e66e57ff9503-kube-api-access-l75rm\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.882083 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c311b6e-3312-4db0-9623-7e4b662e08ca-cert\") pod \"ingress-canary-chwd4\" (UID: \"7c311b6e-3312-4db0-9623-7e4b662e08ca\") " pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.882083 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.882083 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881814 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5f17aa-ec97-43cc-b367-e66e57ff9503-tmp-dir\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.882083 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.881987 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5f17aa-ec97-43cc-b367-e66e57ff9503-config-volume\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.882083 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:09.882058 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:09.882275 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:09.882116 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:10.382088824 +0000 UTC m=+37.008286392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:09.882275 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.882142 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d5f17aa-ec97-43cc-b367-e66e57ff9503-tmp-dir\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.886798 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.886680 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c311b6e-3312-4db0-9623-7e4b662e08ca-cert\") pod \"ingress-canary-chwd4\" (UID: \"7c311b6e-3312-4db0-9623-7e4b662e08ca\") " pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.895203 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.895179 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l75rm\" (UniqueName: \"kubernetes.io/projected/7d5f17aa-ec97-43cc-b367-e66e57ff9503-kube-api-access-l75rm\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:09.896036 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.896013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbxg\" (UniqueName: \"kubernetes.io/projected/7c311b6e-3312-4db0-9623-7e4b662e08ca-kube-api-access-6mbxg\") pod \"ingress-canary-chwd4\" (UID: \"7c311b6e-3312-4db0-9623-7e4b662e08ca\") " pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:09.992071 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:09.992035 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-chwd4" Sep 10 12:31:10.320310 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.320256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-chwd4"] Sep 10 12:31:10.386139 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.386114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:10.386289 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:10.386240 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:10.386347 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:10.386311 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:11.38629079 +0000 UTC m=+38.012488360 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:10.832707 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.832678 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:10.832900 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.832793 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:10.832900 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.832832 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:10.838886 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.838864 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 12:31:10.839679 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.839662 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 12:31:10.839771 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.839664 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 12:31:10.839833 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.839765 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 12:31:10.839884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.839842 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bqff6\"" Sep 10 12:31:10.839937 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:10.839883 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pkls5\"" Sep 10 12:31:11.395191 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:11.395155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:11.395969 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:11.395326 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:11.395969 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:11.395401 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:13.395377897 +0000 UTC m=+40.021575473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:11.673986 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:11.673923 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c311b6e_3312_4db0_9623_7e4b662e08ca.slice/crio-653c7cc40889eb02546df284da97c092f34ccbc6886c672fbe8fa26ecc2ecaec WatchSource:0}: Error finding container 653c7cc40889eb02546df284da97c092f34ccbc6886c672fbe8fa26ecc2ecaec: Status 404 returned error can't find the container with id 653c7cc40889eb02546df284da97c092f34ccbc6886c672fbe8fa26ecc2ecaec Sep 10 12:31:12.060434 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:12.060403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-chwd4" event={"ID":"7c311b6e-3312-4db0-9623-7e4b662e08ca","Type":"ContainerStarted","Data":"653c7cc40889eb02546df284da97c092f34ccbc6886c672fbe8fa26ecc2ecaec"} Sep 10 12:31:13.066091 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:13.066059 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd98d133-19c5-416b-ba3d-ef46bf070e48" containerID="8332c22e33f3f990594c062c7337e0efe61a12cfd6e79676a907591b41b2d4ae" exitCode=0 Sep 10 12:31:13.066519 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:13.066123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerDied","Data":"8332c22e33f3f990594c062c7337e0efe61a12cfd6e79676a907591b41b2d4ae"} Sep 10 12:31:13.408972 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:13.408906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:13.409097 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:13.409079 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:13.409151 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:13.409141 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:17.409125674 +0000 UTC m=+44.035323227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:14.072368 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:14.072165 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd98d133-19c5-416b-ba3d-ef46bf070e48" containerID="6845ad67a8f84db0338c3b86adb2693c9ee8725e67b6feaf9bbd0e49558f7cbf" exitCode=0 Sep 10 12:31:14.072808 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:14.072251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerDied","Data":"6845ad67a8f84db0338c3b86adb2693c9ee8725e67b6feaf9bbd0e49558f7cbf"} Sep 10 12:31:15.076553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:15.076516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-chwd4" event={"ID":"7c311b6e-3312-4db0-9623-7e4b662e08ca","Type":"ContainerStarted","Data":"9250921ed5582d727943161dd9ecbc52c31ee4bf4493796aae4c129b1f86e40e"} Sep 10 12:31:15.080019 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:15.079983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vkghf" event={"ID":"fd98d133-19c5-416b-ba3d-ef46bf070e48","Type":"ContainerStarted","Data":"ef04a65a743c9b50869d32cdd9605f5b62ff497eeaf3858e921ea22c21c01532"} Sep 10 12:31:15.122151 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:15.122059 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vkghf" podStartSLOduration=9.983691246 podStartE2EDuration="41.122042325s" totalImagesPullingTime="31.138351079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:15.120322997 +0000 UTC m=+41.746520577" watchObservedRunningTime="2026-09-10 12:31:15.122042325 +0000 UTC m=+41.748239900" Sep 10 12:31:15.122463 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:15.122438 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-chwd4" podStartSLOduration=2.954551233 podStartE2EDuration="6.122430249s" totalImagesPullingTime="3.167879016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:15.099310705 +0000 UTC m=+41.725508280" watchObservedRunningTime="2026-09-10 12:31:15.122430249 +0000 UTC m=+41.748627833" Sep 10 12:31:17.438913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:17.438860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:17.439439 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:17.439071 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:17.439439 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:17.439183 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:25.439157303 +0000 UTC m=+52.065354900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:25.488771 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:25.488715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:25.489285 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:25.488875 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:25.489285 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:25.488948 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:41.488931247 +0000 UTC m=+68.115128800 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:34.056551 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.056520 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t5558" Sep 10 12:31:34.511333 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.511300 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t"] Sep 10 12:31:34.530707 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.530678 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w"] Sep 10 12:31:34.530844 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.530786 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" Sep 10 12:31:34.532475 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.532457 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2"] Sep 10 12:31:34.532612 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.532592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:34.533464 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.533446 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 10 12:31:34.533612 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.533519 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-64fhv\"" Sep 10 12:31:34.533673 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.533609 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 10 12:31:34.534203 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.534188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.534766 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.534748 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nxqft\"" Sep 10 12:31:34.535525 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.535510 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 10 12:31:34.535570 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.535511 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 10 12:31:34.536434 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.536416 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 10 12:31:34.537880 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.537861 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 12:31:34.538041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.538025 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-stk46\"" Sep 10 12:31:34.538119 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.538092 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 12:31:34.538249 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.538231 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 12:31:34.538340 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.538267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 12:31:34.553932 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.553912 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t"] Sep 10 12:31:34.554913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.554894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2"] Sep 10 12:31:34.557172 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.557152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w"] Sep 10 12:31:34.613113 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.613084 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv"] Sep 10 12:31:34.614820 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.614808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.619277 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.619260 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 10 12:31:34.619734 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.619716 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 10 12:31:34.620312 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.620289 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-648kj\"" Sep 10 12:31:34.620451 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.620427 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 10 12:31:34.623553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.623539 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 10 12:31:34.639375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.639353 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55b7dccb46-4lwvm"] Sep 10 12:31:34.641038 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.641024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.643179 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.643158 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wbn\" (UniqueName: \"kubernetes.io/projected/4153d096-01d3-4be2-836b-e24a797444ef-kube-api-access-g5wbn\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:34.643244 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.643202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvb2\" (UniqueName: \"kubernetes.io/projected/e6855898-d009-4ae4-a5b1-b44df64edb60-kube-api-access-jtvb2\") pod \"volume-data-source-validator-695c895bd6-lj58t\" (UID: \"e6855898-d009-4ae4-a5b1-b44df64edb60\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" Sep 10 12:31:34.643281 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.643245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:34.643315 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.643283 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93b9099d-bd81-4d89-a0d8-76a95392a2a7-serving-cert\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.643347 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.643322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmrn\" (UniqueName: \"kubernetes.io/projected/93b9099d-bd81-4d89-a0d8-76a95392a2a7-kube-api-access-2pmrn\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.643347 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.643339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93b9099d-bd81-4d89-a0d8-76a95392a2a7-config\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.648502 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.648479 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-137-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" Sep 10 12:31:34.648604 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.648549 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-137-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" Sep 10 12:31:34.648604 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.648556 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-137-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" Sep 10 12:31:34.648604 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.648574 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-957dd\" is forbidden: User \"system:node:ip-10-0-137-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-957dd\"" type="*v1.Secret" Sep 10 12:31:34.648722 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.648614 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-137-89.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-137-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" Sep 10 12:31:34.650402 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.650385 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv"] Sep 10 12:31:34.669374 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.669353 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b7dccb46-4lwvm"] Sep 10 12:31:34.744310 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.744395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wbn\" (UniqueName: \"kubernetes.io/projected/4153d096-01d3-4be2-836b-e24a797444ef-kube-api-access-g5wbn\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:34.744395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.744395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-bound-sa-token\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.744492 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744470 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvb2\" (UniqueName: \"kubernetes.io/projected/e6855898-d009-4ae4-a5b1-b44df64edb60-kube-api-access-jtvb2\") pod \"volume-data-source-validator-695c895bd6-lj58t\" (UID: \"e6855898-d009-4ae4-a5b1-b44df64edb60\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" Sep 10 12:31:34.744572 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:34.744648 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-trusted-ca\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.744648 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744633 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-installation-pull-secrets\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.744720 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.744720 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.744676 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 12:31:34.744800 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:34.744729 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls podName:4153d096-01d3-4be2-836b-e24a797444ef nodeName:}" failed. No retries permitted until 2026-09-10 12:31:35.24471588 +0000 UTC m=+61.870913434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sb76w" (UID: "4153d096-01d3-4be2-836b-e24a797444ef") : secret "samples-operator-tls" not found Sep 10 12:31:34.744800 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93b9099d-bd81-4d89-a0d8-76a95392a2a7-serving-cert\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.744877 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w75sb\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-kube-api-access-w75sb\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.744877 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.744974 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744884 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q25cf\" (UniqueName: \"kubernetes.io/projected/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-kube-api-access-q25cf\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.744974 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d8784e-6286-4b16-a293-d0ba76a79afb-ca-trust-extracted\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.745074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.744979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmrn\" (UniqueName: \"kubernetes.io/projected/93b9099d-bd81-4d89-a0d8-76a95392a2a7-kube-api-access-2pmrn\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.745074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.745016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-certificates\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.745074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.745049 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93b9099d-bd81-4d89-a0d8-76a95392a2a7-config\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.745516 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.745497 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/93b9099d-bd81-4d89-a0d8-76a95392a2a7-config\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.747436 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.747418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/93b9099d-bd81-4d89-a0d8-76a95392a2a7-serving-cert\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.755999 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.755976 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wbn\" (UniqueName: \"kubernetes.io/projected/4153d096-01d3-4be2-836b-e24a797444ef-kube-api-access-g5wbn\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:34.756192 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.756172 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmrn\" (UniqueName: \"kubernetes.io/projected/93b9099d-bd81-4d89-a0d8-76a95392a2a7-kube-api-access-2pmrn\") pod \"service-ca-operator-859f94c4b-l2dc2\" (UID: \"93b9099d-bd81-4d89-a0d8-76a95392a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.757138 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.757122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvb2\" (UniqueName: \"kubernetes.io/projected/e6855898-d009-4ae4-a5b1-b44df64edb60-kube-api-access-jtvb2\") pod \"volume-data-source-validator-695c895bd6-lj58t\" (UID: \"e6855898-d009-4ae4-a5b1-b44df64edb60\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" Sep 10 12:31:34.841023 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.840971 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" Sep 10 12:31:34.846020 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846001 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q25cf\" (UniqueName: \"kubernetes.io/projected/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-kube-api-access-q25cf\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.846124 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d8784e-6286-4b16-a293-d0ba76a79afb-ca-trust-extracted\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846124 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-certificates\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846389 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-bound-sa-token\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846389 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846271 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-trusted-ca\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846389 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-installation-pull-secrets\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846389 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.846664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d8784e-6286-4b16-a293-d0ba76a79afb-ca-trust-extracted\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w75sb\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-kube-api-access-w75sb\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.846664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.846783 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.846742 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-certificates\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.847274 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.847250 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.848435 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.848414 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.850421 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.850401 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" Sep 10 12:31:34.859630 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.859576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q25cf\" (UniqueName: \"kubernetes.io/projected/88e683cb-1f68-45b1-bcd9-a85bb66c9e1c-kube-api-access-q25cf\") pod \"kube-storage-version-migrator-operator-8d7dd4675-rtqmv\" (UID: \"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:34.861571 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.861543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w75sb\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-kube-api-access-w75sb\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.861681 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.861604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-bound-sa-token\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:34.923000 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:34.922794 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" Sep 10 12:31:35.124748 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.124680 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t"] Sep 10 12:31:35.128229 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:35.128203 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6855898_d009_4ae4_a5b1_b44df64edb60.slice/crio-a4307a6f82e2570adb86b5f9479bb143b1c2a0be754099d9c145bf286b4b8aa6 WatchSource:0}: Error finding container a4307a6f82e2570adb86b5f9479bb143b1c2a0be754099d9c145bf286b4b8aa6: Status 404 returned error can't find the container with id a4307a6f82e2570adb86b5f9479bb143b1c2a0be754099d9c145bf286b4b8aa6 Sep 10 12:31:35.151130 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.151103 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2"] Sep 10 12:31:35.155374 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:35.155346 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b9099d_bd81_4d89_a0d8_76a95392a2a7.slice/crio-06f6e23e0760c7c24b3ea6065e71330c83454ce526bc2dafd61a9f0bb5c97ea2 WatchSource:0}: Error finding container 06f6e23e0760c7c24b3ea6065e71330c83454ce526bc2dafd61a9f0bb5c97ea2: Status 404 returned error can't find the container with id 06f6e23e0760c7c24b3ea6065e71330c83454ce526bc2dafd61a9f0bb5c97ea2 Sep 10 12:31:35.197428 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.197407 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv"] Sep 10 12:31:35.200515 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:35.200496 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e683cb_1f68_45b1_bcd9_a85bb66c9e1c.slice/crio-122a8e1fe556f7e175a4874f23a1223b3ce1ff12ed5cfcddc293ffd0f18120e1 WatchSource:0}: Error finding container 122a8e1fe556f7e175a4874f23a1223b3ce1ff12ed5cfcddc293ffd0f18120e1: Status 404 returned error can't find the container with id 122a8e1fe556f7e175a4874f23a1223b3ce1ff12ed5cfcddc293ffd0f18120e1 Sep 10 12:31:35.250222 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.250196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:35.250363 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.250345 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 12:31:35.250405 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.250400 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls podName:4153d096-01d3-4be2-836b-e24a797444ef nodeName:}" failed. No retries permitted until 2026-09-10 12:31:36.250386358 +0000 UTC m=+62.876583911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sb76w" (UID: "4153d096-01d3-4be2-836b-e24a797444ef") : secret "samples-operator-tls" not found Sep 10 12:31:35.644432 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.644399 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 12:31:35.649178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.649154 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-installation-pull-secrets\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:35.707102 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.707075 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 12:31:35.717656 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.717634 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-trusted-ca\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:35.793059 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.793036 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-957dd\"" Sep 10 12:31:35.826187 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.826171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 12:31:35.826375 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.826358 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 12:31:35.826421 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.826378 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55b7dccb46-4lwvm: secret "image-registry-tls" not found Sep 10 12:31:35.826456 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.826429 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls podName:49d8784e-6286-4b16-a293-d0ba76a79afb nodeName:}" failed. No retries permitted until 2026-09-10 12:31:36.326409902 +0000 UTC m=+62.952607474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls") pod "image-registry-55b7dccb46-4lwvm" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb") : secret "image-registry-tls" not found Sep 10 12:31:35.846408 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.846387 2562 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: failed to sync secret cache: timed out waiting for the condition Sep 10 12:31:35.846502 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:35.846445 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration podName:49d8784e-6286-4b16-a293-d0ba76a79afb nodeName:}" failed. No retries permitted until 2026-09-10 12:31:36.346432323 +0000 UTC m=+62.972629879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration") pod "image-registry-55b7dccb46-4lwvm" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb") : failed to sync secret cache: timed out waiting for the condition Sep 10 12:31:35.934696 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:35.934635 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 12:31:36.132787 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.132748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" event={"ID":"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c","Type":"ContainerStarted","Data":"122a8e1fe556f7e175a4874f23a1223b3ce1ff12ed5cfcddc293ffd0f18120e1"} Sep 10 12:31:36.134151 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.134120 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" event={"ID":"e6855898-d009-4ae4-a5b1-b44df64edb60","Type":"ContainerStarted","Data":"a4307a6f82e2570adb86b5f9479bb143b1c2a0be754099d9c145bf286b4b8aa6"} Sep 10 12:31:36.135685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.135656 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" event={"ID":"93b9099d-bd81-4d89-a0d8-76a95392a2a7","Type":"ContainerStarted","Data":"06f6e23e0760c7c24b3ea6065e71330c83454ce526bc2dafd61a9f0bb5c97ea2"} Sep 10 12:31:36.259287 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.259248 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:36.259527 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:36.259501 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 12:31:36.259690 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:36.259568 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls podName:4153d096-01d3-4be2-836b-e24a797444ef nodeName:}" failed. No retries permitted until 2026-09-10 12:31:38.25955063 +0000 UTC m=+64.885748196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sb76w" (UID: "4153d096-01d3-4be2-836b-e24a797444ef") : secret "samples-operator-tls" not found Sep 10 12:31:36.360107 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.360071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:36.360287 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.360136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:36.360287 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:36.360261 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 12:31:36.360287 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:36.360276 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55b7dccb46-4lwvm: secret "image-registry-tls" not found Sep 10 12:31:36.360421 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:36.360337 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls podName:49d8784e-6286-4b16-a293-d0ba76a79afb nodeName:}" failed. No retries permitted until 2026-09-10 12:31:37.360319476 +0000 UTC m=+63.986517045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls") pod "image-registry-55b7dccb46-4lwvm" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb") : secret "image-registry-tls" not found Sep 10 12:31:36.363196 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:36.363165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:37.140908 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:37.140820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" event={"ID":"e6855898-d009-4ae4-a5b1-b44df64edb60","Type":"ContainerStarted","Data":"596ebf3a3d5f18be52b17105f2269aa84ac5be4e798493a611cae4252fa04647"} Sep 10 12:31:37.163297 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:37.163238 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-lj58t" podStartSLOduration=1.448410939 podStartE2EDuration="3.163217744s" totalImagesPullingTime="1.714806805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:37.16203529 +0000 UTC m=+63.788232866" watchObservedRunningTime="2026-09-10 12:31:37.163217744 +0000 UTC m=+63.789415320" Sep 10 12:31:37.366955 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:37.366919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:37.367129 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:37.367081 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 12:31:37.367129 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:37.367105 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55b7dccb46-4lwvm: secret "image-registry-tls" not found Sep 10 12:31:37.367262 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:37.367172 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls podName:49d8784e-6286-4b16-a293-d0ba76a79afb nodeName:}" failed. No retries permitted until 2026-09-10 12:31:39.367153534 +0000 UTC m=+65.993351089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls") pod "image-registry-55b7dccb46-4lwvm" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb") : secret "image-registry-tls" not found Sep 10 12:31:38.144960 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.144882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" event={"ID":"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c","Type":"ContainerStarted","Data":"5fd694754eaa1d1c42ce8d627f61c934f9e992cd5bf84fc0c986a3c9a4420842"} Sep 10 12:31:38.146786 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.146756 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" event={"ID":"93b9099d-bd81-4d89-a0d8-76a95392a2a7","Type":"ContainerStarted","Data":"cd0c97cfef216d77555e744cb2fe9b11c675bd324156c8bb9affe0e1f874cc45"} Sep 10 12:31:38.168725 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.168678 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" podStartSLOduration=1.4916022789999999 podStartE2EDuration="4.168665429s" totalImagesPullingTime="2.67706315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:38.166023733 +0000 UTC m=+64.792221308" watchObservedRunningTime="2026-09-10 12:31:38.168665429 +0000 UTC m=+64.794863003" Sep 10 12:31:38.189795 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.189731 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" podStartSLOduration=1.468442325 podStartE2EDuration="4.189712622s" totalImagesPullingTime="2.721270297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:38.189175772 +0000 UTC m=+64.815373349" watchObservedRunningTime="2026-09-10 12:31:38.189712622 +0000 UTC m=+64.815910198" Sep 10 12:31:38.275289 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.275045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:38.275289 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:38.275183 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 12:31:38.275289 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:38.275255 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls podName:4153d096-01d3-4be2-836b-e24a797444ef nodeName:}" failed. No retries permitted until 2026-09-10 12:31:42.275234491 +0000 UTC m=+68.901432049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sb76w" (UID: "4153d096-01d3-4be2-836b-e24a797444ef") : secret "samples-operator-tls" not found Sep 10 12:31:38.897181 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.897151 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc"] Sep 10 12:31:38.899429 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.899410 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" Sep 10 12:31:38.902731 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.902703 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 12:31:38.902731 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.902714 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 12:31:38.902885 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.902846 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-s7m72\"" Sep 10 12:31:38.909857 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:38.909836 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc"] Sep 10 12:31:39.082612 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.082568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvp6\" (UniqueName: \"kubernetes.io/projected/a0b64a56-a956-41c8-a0df-24209006a2ea-kube-api-access-6gvp6\") pod \"migrator-fdbb8fdff-svhwc\" (UID: \"a0b64a56-a956-41c8-a0df-24209006a2ea\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" Sep 10 12:31:39.183355 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.183273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvp6\" (UniqueName: \"kubernetes.io/projected/a0b64a56-a956-41c8-a0df-24209006a2ea-kube-api-access-6gvp6\") pod \"migrator-fdbb8fdff-svhwc\" (UID: \"a0b64a56-a956-41c8-a0df-24209006a2ea\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" Sep 10 12:31:39.198237 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.198209 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvp6\" (UniqueName: \"kubernetes.io/projected/a0b64a56-a956-41c8-a0df-24209006a2ea-kube-api-access-6gvp6\") pod \"migrator-fdbb8fdff-svhwc\" (UID: \"a0b64a56-a956-41c8-a0df-24209006a2ea\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" Sep 10 12:31:39.208069 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.208051 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" Sep 10 12:31:39.385143 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.385113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:39.385289 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.385238 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 12:31:39.385289 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.385252 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55b7dccb46-4lwvm: secret "image-registry-tls" not found Sep 10 12:31:39.385359 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.385304 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls podName:49d8784e-6286-4b16-a293-d0ba76a79afb nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.385288593 +0000 UTC m=+70.011486158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls") pod "image-registry-55b7dccb46-4lwvm" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb") : secret "image-registry-tls" not found Sep 10 12:31:39.474847 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.474823 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc"] Sep 10 12:31:39.477058 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:39.477033 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b64a56_a956_41c8_a0df_24209006a2ea.slice/crio-f702ab82578e91e0ae789269a85d9c2f47fda1c9ca544663f81ce39279121c92 WatchSource:0}: Error finding container f702ab82578e91e0ae789269a85d9c2f47fda1c9ca544663f81ce39279121c92: Status 404 returned error can't find the container with id f702ab82578e91e0ae789269a85d9c2f47fda1c9ca544663f81ce39279121c92 Sep 10 12:31:39.586667 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.586641 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:39.586775 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.586739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:31:39.590695 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.590680 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 12:31:39.590853 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.590839 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 12:31:39.597544 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.597528 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 12:31:39.597609 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.597598 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs podName:3c49d065-9a7d-4b66-b420-0bd16b7dfbae nodeName:}" failed. No retries permitted until 2026-09-10 12:32:43.597563895 +0000 UTC m=+130.223761449 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs") pod "network-metrics-daemon-6s69q" (UID: "3c49d065-9a7d-4b66-b420-0bd16b7dfbae") : secret "metrics-daemon-secret" not found Sep 10 12:31:39.599791 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.599773 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/46be07f1-b76e-451e-a75e-0ac2af42c40a-original-pull-secret\") pod \"global-pull-secret-syncer-5s256\" (UID: \"46be07f1-b76e-451e-a75e-0ac2af42c40a\") " pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:39.629111 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.629092 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mpxpr"] Sep 10 12:31:39.632060 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.632047 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.635631 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.635614 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 12:31:39.635783 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.635766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r87vl\"" Sep 10 12:31:39.636048 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.636034 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 12:31:39.636122 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.636096 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 12:31:39.636333 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.636318 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 12:31:39.643300 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.643280 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5s256" Sep 10 12:31:39.644203 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.644183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpxpr"] Sep 10 12:31:39.688098 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.688065 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:39.691362 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.690950 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 12:31:39.701312 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.701290 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 12:31:39.712575 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.712556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpkg\" (UniqueName: \"kubernetes.io/projected/1110380b-db68-445a-9601-1f6dedd8c1c9-kube-api-access-9bpkg\") pod \"network-check-target-twpc8\" (UID: \"1110380b-db68-445a-9601-1f6dedd8c1c9\") " pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:39.788626 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.788604 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-data-volume\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.788725 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.788661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.788725 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.788710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-crio-socket\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.788822 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.788734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzgk\" (UniqueName: \"kubernetes.io/projected/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-api-access-wlzgk\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.788822 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.788759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889165 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzgk\" (UniqueName: \"kubernetes.io/projected/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-api-access-wlzgk\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889265 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889265 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-data-volume\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889374 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.889321 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 12:31:39.889423 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889423 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.889386 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:40.389364208 +0000 UTC m=+67.015561781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : secret "insights-runtime-extractor-tls" not found Sep 10 12:31:39.889521 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.889424 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 12:31:39.889521 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-crio-socket\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889521 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-data-volume\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.889521 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:39.889483 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:40.389467724 +0000 UTC m=+67.015665296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : configmap "kube-rbac-proxy" not found Sep 10 12:31:39.889687 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.889533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-crio-socket\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.898428 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.898411 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzgk\" (UniqueName: \"kubernetes.io/projected/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-api-access-wlzgk\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:39.908370 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.908348 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5s256"] Sep 10 12:31:39.911901 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:39.911884 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46be07f1_b76e_451e_a75e_0ac2af42c40a.slice/crio-0ffed0559408a0c55a0356721d2b2ce24af323ecf5e38ae38568f13e635ac46a WatchSource:0}: Error finding container 0ffed0559408a0c55a0356721d2b2ce24af323ecf5e38ae38568f13e635ac46a: Status 404 returned error can't find the container with id 0ffed0559408a0c55a0356721d2b2ce24af323ecf5e38ae38568f13e635ac46a Sep 10 12:31:39.952394 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.952376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pkls5\"" Sep 10 12:31:39.960646 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:39.960628 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:40.153347 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:40.153269 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5s256" event={"ID":"46be07f1-b76e-451e-a75e-0ac2af42c40a","Type":"ContainerStarted","Data":"0ffed0559408a0c55a0356721d2b2ce24af323ecf5e38ae38568f13e635ac46a"} Sep 10 12:31:40.154433 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:40.154412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" event={"ID":"a0b64a56-a956-41c8-a0df-24209006a2ea","Type":"ContainerStarted","Data":"f702ab82578e91e0ae789269a85d9c2f47fda1c9ca544663f81ce39279121c92"} Sep 10 12:31:40.234086 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:40.234057 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twpc8"] Sep 10 12:31:40.236774 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:40.236744 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1110380b_db68_445a_9601_1f6dedd8c1c9.slice/crio-f584f2ccf16d84e0dee507b5b5f65bf6cd0960df8567eede3746f43b6fd88a05 WatchSource:0}: Error finding container f584f2ccf16d84e0dee507b5b5f65bf6cd0960df8567eede3746f43b6fd88a05: Status 404 returned error can't find the container with id f584f2ccf16d84e0dee507b5b5f65bf6cd0960df8567eede3746f43b6fd88a05 Sep 10 12:31:40.394710 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:40.394680 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:40.394860 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:40.394747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:40.394860 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:40.394843 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 12:31:40.394953 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:40.394896 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:41.394883222 +0000 UTC m=+68.021080775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : secret "insights-runtime-extractor-tls" not found Sep 10 12:31:40.394953 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:40.394843 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 12:31:40.395033 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:40.394959 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:41.394948517 +0000 UTC m=+68.021146089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : configmap "kube-rbac-proxy" not found Sep 10 12:31:41.160167 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:41.159974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twpc8" event={"ID":"1110380b-db68-445a-9601-1f6dedd8c1c9","Type":"ContainerStarted","Data":"f584f2ccf16d84e0dee507b5b5f65bf6cd0960df8567eede3746f43b6fd88a05"} Sep 10 12:31:41.404076 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:41.404025 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:41.404494 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:41.404140 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:41.404494 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:41.404263 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 12:31:41.404494 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:41.404327 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.404308331 +0000 UTC m=+70.030505909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : secret "insights-runtime-extractor-tls" not found Sep 10 12:31:41.404720 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:41.404707 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 12:31:41.404768 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:41.404756 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.404740825 +0000 UTC m=+70.030938395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : configmap "kube-rbac-proxy" not found Sep 10 12:31:41.504731 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:41.504702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:31:41.504910 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:41.504849 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 12:31:41.504980 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:41.504916 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls podName:7d5f17aa-ec97-43cc-b367-e66e57ff9503 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:13.50489774 +0000 UTC m=+100.131095312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls") pod "dns-default-jjtjh" (UID: "7d5f17aa-ec97-43cc-b367-e66e57ff9503") : secret "dns-default-metrics-tls" not found Sep 10 12:31:42.167174 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:42.167138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" event={"ID":"a0b64a56-a956-41c8-a0df-24209006a2ea","Type":"ContainerStarted","Data":"2e9afdd4d0fedeacda65253b7a24fb6517b3228dd2cbb39adf8d57e599e14599"} Sep 10 12:31:42.167318 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:42.167206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" event={"ID":"a0b64a56-a956-41c8-a0df-24209006a2ea","Type":"ContainerStarted","Data":"c19caf2b72e867f861c334bfa182b6db6336ec9db85aea1d62ef82192ca89682"} Sep 10 12:31:42.185787 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:42.185729 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-svhwc" podStartSLOduration=2.145369146 podStartE2EDuration="4.185711901s" totalImagesPullingTime="2.040342755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:42.185339383 +0000 UTC m=+68.811536972" watchObservedRunningTime="2026-09-10 12:31:42.185711901 +0000 UTC m=+68.811909476" Sep 10 12:31:42.311745 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:42.311705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:42.311903 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:42.311864 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 10 12:31:42.311961 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:42.311935 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls podName:4153d096-01d3-4be2-836b-e24a797444ef nodeName:}" failed. No retries permitted until 2026-09-10 12:31:50.311915191 +0000 UTC m=+76.938112747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-sb76w" (UID: "4153d096-01d3-4be2-836b-e24a797444ef") : secret "samples-operator-tls" not found Sep 10 12:31:42.369451 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:42.369424 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5kcqc_57937ad4-a410-4961-8ba1-e72addf6c5a3/dns-node-resolver/0.log" Sep 10 12:31:43.417977 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:43.417943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:43.418017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:43.418049 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418087 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418158 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418189 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418165 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:47.418142898 +0000 UTC m=+74.044340490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : configmap "kube-rbac-proxy" not found Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418210 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55b7dccb46-4lwvm: secret "image-registry-tls" not found Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418225 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:47.418211167 +0000 UTC m=+74.044408725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : secret "insights-runtime-extractor-tls" not found Sep 10 12:31:43.418371 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:43.418256 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls podName:49d8784e-6286-4b16-a293-d0ba76a79afb nodeName:}" failed. No retries permitted until 2026-09-10 12:31:51.418240681 +0000 UTC m=+78.044438241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls") pod "image-registry-55b7dccb46-4lwvm" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb") : secret "image-registry-tls" not found Sep 10 12:31:43.571394 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:43.571367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zz8th_9a6dfcda-b4c5-41c0-9c31-53995fb29b8c/node-ca/0.log" Sep 10 12:31:44.168503 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:44.168468 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chwd4_7c311b6e-3312-4db0-9623-7e4b662e08ca/serve-healthcheck-canary/0.log" Sep 10 12:31:44.574468 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:44.574435 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-svhwc_a0b64a56-a956-41c8-a0df-24209006a2ea/migrator/0.log" Sep 10 12:31:44.768634 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:44.768607 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-svhwc_a0b64a56-a956-41c8-a0df-24209006a2ea/graceful-termination/0.log" Sep 10 12:31:44.972287 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:44.972254 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rtqmv_88e683cb-1f68-45b1-bcd9-a85bb66c9e1c/kube-storage-version-migrator-operator/0.log" Sep 10 12:31:45.178809 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:45.178724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5s256" event={"ID":"46be07f1-b76e-451e-a75e-0ac2af42c40a","Type":"ContainerStarted","Data":"108b610fdf67d33485144f5a722bb9d4d2b94b924bad49a4c7930c0bc3df2e81"} Sep 10 12:31:45.180133 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:45.180107 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twpc8" event={"ID":"1110380b-db68-445a-9601-1f6dedd8c1c9","Type":"ContainerStarted","Data":"7af99ce7bb9ed11cdaf746fb99b947201336a4208b6b2a7b9378a3cc1347aeec"} Sep 10 12:31:45.180266 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:45.180229 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:31:45.221101 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:45.221059 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5s256" podStartSLOduration=66.388074857 podStartE2EDuration="1m11.221047669s" totalImagesPullingTime="4.832972812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:45.219620662 +0000 UTC m=+71.845818239" watchObservedRunningTime="2026-09-10 12:31:45.221047669 +0000 UTC m=+71.847245243" Sep 10 12:31:45.245803 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:45.245762 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-twpc8" podStartSLOduration=66.741876851 podStartE2EDuration="1m11.245748603s" totalImagesPullingTime="4.503871752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:45.244847001 +0000 UTC m=+71.871044576" watchObservedRunningTime="2026-09-10 12:31:45.245748603 +0000 UTC m=+71.871946178" Sep 10 12:31:47.447599 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:47.447547 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:47.448042 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:47.447652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:47.448042 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:47.447672 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 12:31:47.448042 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:47.447729 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:55.447715259 +0000 UTC m=+82.073912813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : configmap "kube-rbac-proxy" not found Sep 10 12:31:47.448042 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:47.447751 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 12:31:47.448042 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:47.447799 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:55.447783703 +0000 UTC m=+82.073981279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : secret "insights-runtime-extractor-tls" not found Sep 10 12:31:50.371212 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:50.371154 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:50.374526 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:50.374498 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4153d096-01d3-4be2-836b-e24a797444ef-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-sb76w\" (UID: \"4153d096-01d3-4be2-836b-e24a797444ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:50.446684 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:50.446651 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" Sep 10 12:31:50.719250 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:50.719178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w"] Sep 10 12:31:51.199702 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:51.199662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" event={"ID":"4153d096-01d3-4be2-836b-e24a797444ef","Type":"ContainerStarted","Data":"1b77b4277d148f366866227919fd322dd203b495802f70fd8abc7036386e1892"} Sep 10 12:31:51.480480 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:51.480409 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:51.483015 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:51.482993 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"image-registry-55b7dccb46-4lwvm\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:51.748286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:51.748204 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:52.033349 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:52.033324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55b7dccb46-4lwvm"] Sep 10 12:31:52.035830 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:31:52.035789 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d8784e_6286_4b16_a293_d0ba76a79afb.slice/crio-a131e27228286bd5ae48e221224a3c378af86012c0ec2ab1365a5003456d30db WatchSource:0}: Error finding container a131e27228286bd5ae48e221224a3c378af86012c0ec2ab1365a5003456d30db: Status 404 returned error can't find the container with id a131e27228286bd5ae48e221224a3c378af86012c0ec2ab1365a5003456d30db Sep 10 12:31:52.206878 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:52.206842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" event={"ID":"49d8784e-6286-4b16-a293-d0ba76a79afb","Type":"ContainerStarted","Data":"a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc"} Sep 10 12:31:52.207027 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:52.206902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" event={"ID":"49d8784e-6286-4b16-a293-d0ba76a79afb","Type":"ContainerStarted","Data":"a131e27228286bd5ae48e221224a3c378af86012c0ec2ab1365a5003456d30db"} Sep 10 12:31:52.207027 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:52.206940 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:31:52.238034 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:52.237981 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" podStartSLOduration=18.237959902 podStartE2EDuration="18.237959902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:52.236903066 +0000 UTC m=+78.863100642" watchObservedRunningTime="2026-09-10 12:31:52.237959902 +0000 UTC m=+78.864157479" Sep 10 12:31:53.211741 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:53.211708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" event={"ID":"4153d096-01d3-4be2-836b-e24a797444ef","Type":"ContainerStarted","Data":"08d153dd1670e8d1b6c53f250d9d7352b88314ab276f04b5bd28b6c99ac6998d"} Sep 10 12:31:53.212074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:53.211768 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" event={"ID":"4153d096-01d3-4be2-836b-e24a797444ef","Type":"ContainerStarted","Data":"a908418106a8081d0e1e47c4e51b1ce7dba0392038034445a2714fa18e479f15"} Sep 10 12:31:55.509805 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:55.509671 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:55.509805 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:55.509757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:31:55.510356 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:55.509893 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 12:31:55.510356 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:31:55.509965 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls podName:3b03053e-fdd6-4256-8300-cfcd0c67e5f8 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:11.509945537 +0000 UTC m=+98.136143092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mpxpr" (UID: "3b03053e-fdd6-4256-8300-cfcd0c67e5f8") : secret "insights-runtime-extractor-tls" not found Sep 10 12:31:55.510356 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:31:55.510151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:32:07.186320 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:07.186266 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-sb76w" podStartSLOduration=30.935545589 podStartE2EDuration="33.186248973s" totalImagesPullingTime="2.250703384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:53.237177798 +0000 UTC m=+79.863375372" watchObservedRunningTime="2026-09-10 12:32:07.186248973 +0000 UTC m=+93.812446548" Sep 10 12:32:07.186813 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:07.186509 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55b7dccb46-4lwvm"] Sep 10 12:32:07.190751 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:07.190723 2562 patch_prober.go:28] interesting pod/image-registry-55b7dccb46-4lwvm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 12:32:07.190876 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:07.190767 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" podUID="49d8784e-6286-4b16-a293-d0ba76a79afb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 12:32:09.843227 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:09.843198 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np"] Sep 10 12:32:09.876328 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:09.876303 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np"] Sep 10 12:32:09.876469 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:09.876342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:09.879671 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:09.879654 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 12:32:09.879798 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:09.879718 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-252w9\"" Sep 10 12:32:09.911935 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:09.911912 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2f156b43-9250-40af-8193-d3300a188afc-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vn7np\" (UID: \"2f156b43-9250-40af-8193-d3300a188afc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:10.012657 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:10.012624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2f156b43-9250-40af-8193-d3300a188afc-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vn7np\" (UID: \"2f156b43-9250-40af-8193-d3300a188afc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:10.012789 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:32:10.012760 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 10 12:32:10.012841 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:32:10.012830 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f156b43-9250-40af-8193-d3300a188afc-tls-certificates podName:2f156b43-9250-40af-8193-d3300a188afc nodeName:}" failed. No retries permitted until 2026-09-10 12:32:10.512814931 +0000 UTC m=+97.139012485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/2f156b43-9250-40af-8193-d3300a188afc-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-vn7np" (UID: "2f156b43-9250-40af-8193-d3300a188afc") : secret "prometheus-operator-admission-webhook-tls" not found Sep 10 12:32:10.515878 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:10.515840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2f156b43-9250-40af-8193-d3300a188afc-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vn7np\" (UID: \"2f156b43-9250-40af-8193-d3300a188afc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:10.518456 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:10.518430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2f156b43-9250-40af-8193-d3300a188afc-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-vn7np\" (UID: \"2f156b43-9250-40af-8193-d3300a188afc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:10.785209 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:10.785124 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:11.056331 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:11.056263 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np"] Sep 10 12:32:11.059723 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:11.059697 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f156b43_9250_40af_8193_d3300a188afc.slice/crio-2cea2212365fbe505787db1ab2e66a79de0b38de92c97c7643781600899818ec WatchSource:0}: Error finding container 2cea2212365fbe505787db1ab2e66a79de0b38de92c97c7643781600899818ec: Status 404 returned error can't find the container with id 2cea2212365fbe505787db1ab2e66a79de0b38de92c97c7643781600899818ec Sep 10 12:32:11.274371 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:11.274344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" event={"ID":"2f156b43-9250-40af-8193-d3300a188afc","Type":"ContainerStarted","Data":"2cea2212365fbe505787db1ab2e66a79de0b38de92c97c7643781600899818ec"} Sep 10 12:32:11.523181 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:11.523156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:32:11.525789 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:11.525767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b03053e-fdd6-4256-8300-cfcd0c67e5f8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mpxpr\" (UID: \"3b03053e-fdd6-4256-8300-cfcd0c67e5f8\") " pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:32:11.740510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:11.740471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mpxpr" Sep 10 12:32:12.014394 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:12.014361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mpxpr"] Sep 10 12:32:12.017832 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:12.017805 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b03053e_fdd6_4256_8300_cfcd0c67e5f8.slice/crio-7815e493ddff2e246c60aefb85e1e716e1a0e727ee0cb86e6e9d6580dc9b84ba WatchSource:0}: Error finding container 7815e493ddff2e246c60aefb85e1e716e1a0e727ee0cb86e6e9d6580dc9b84ba: Status 404 returned error can't find the container with id 7815e493ddff2e246c60aefb85e1e716e1a0e727ee0cb86e6e9d6580dc9b84ba Sep 10 12:32:12.280029 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:12.279995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpxpr" event={"ID":"3b03053e-fdd6-4256-8300-cfcd0c67e5f8","Type":"ContainerStarted","Data":"7f3838e0ae330359abcc31ea7a5db7f829739068ec033139a5df0d0b9a922608"} Sep 10 12:32:12.280397 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:12.280057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpxpr" event={"ID":"3b03053e-fdd6-4256-8300-cfcd0c67e5f8","Type":"ContainerStarted","Data":"7815e493ddff2e246c60aefb85e1e716e1a0e727ee0cb86e6e9d6580dc9b84ba"} Sep 10 12:32:13.286157 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.286116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpxpr" event={"ID":"3b03053e-fdd6-4256-8300-cfcd0c67e5f8","Type":"ContainerStarted","Data":"45c2b9bbca606916799c3c78fe86fd8273703a3910a97367075fdce7a1506daf"} Sep 10 12:32:13.287779 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.287751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" event={"ID":"2f156b43-9250-40af-8193-d3300a188afc","Type":"ContainerStarted","Data":"8b8bf9551e806607eb7cc7a50c31288f4ee335f99a2a3d9445bde8d7c4663033"} Sep 10 12:32:13.287984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.287961 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:13.294884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.294450 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" Sep 10 12:32:13.305782 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.305730 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-vn7np" podStartSLOduration=2.885912972 podStartE2EDuration="4.305713201s" totalImagesPullingTime="1.419800229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:13.305031559 +0000 UTC m=+99.931229149" watchObservedRunningTime="2026-09-10 12:32:13.305713201 +0000 UTC m=+99.931910777" Sep 10 12:32:13.537196 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.537115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:32:13.539481 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.539450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7d5f17aa-ec97-43cc-b367-e66e57ff9503-metrics-tls\") pod \"dns-default-jjtjh\" (UID: \"7d5f17aa-ec97-43cc-b367-e66e57ff9503\") " pod="openshift-dns/dns-default-jjtjh" Sep 10 12:32:13.580139 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.580115 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c86jl\"" Sep 10 12:32:13.588228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.588204 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jjtjh" Sep 10 12:32:13.873627 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:13.873539 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jjtjh"] Sep 10 12:32:14.148406 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:14.148337 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5f17aa_ec97_43cc_b367_e66e57ff9503.slice/crio-f5f78543598a78a9a20cfaab6128ff419f8c237a52b4aedf7d6be1d411ad1ec2 WatchSource:0}: Error finding container f5f78543598a78a9a20cfaab6128ff419f8c237a52b4aedf7d6be1d411ad1ec2: Status 404 returned error can't find the container with id f5f78543598a78a9a20cfaab6128ff419f8c237a52b4aedf7d6be1d411ad1ec2 Sep 10 12:32:14.291947 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:14.291924 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjtjh" event={"ID":"7d5f17aa-ec97-43cc-b367-e66e57ff9503","Type":"ContainerStarted","Data":"f5f78543598a78a9a20cfaab6128ff419f8c237a52b4aedf7d6be1d411ad1ec2"} Sep 10 12:32:14.294109 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:14.294078 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mpxpr" event={"ID":"3b03053e-fdd6-4256-8300-cfcd0c67e5f8","Type":"ContainerStarted","Data":"cfe51f30ef04606c4fb92c5db1b2c957c5dca422f988a77ceb9875d9a2deb24e"} Sep 10 12:32:14.316117 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:14.316072 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mpxpr" podStartSLOduration=33.252935989 podStartE2EDuration="35.316058183s" totalImagesPullingTime="2.063122194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:31:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:14.313841764 +0000 UTC m=+100.940039339" watchObservedRunningTime="2026-09-10 12:32:14.316058183 +0000 UTC m=+100.942255758" Sep 10 12:32:16.185903 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:16.185877 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-twpc8" Sep 10 12:32:16.302613 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:16.302560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjtjh" event={"ID":"7d5f17aa-ec97-43cc-b367-e66e57ff9503","Type":"ContainerStarted","Data":"ddca390cbbf28e0df2f221155519980c51cbc64c269969eeacdc9701406d80e2"} Sep 10 12:32:16.302772 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:16.302657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jjtjh" event={"ID":"7d5f17aa-ec97-43cc-b367-e66e57ff9503","Type":"ContainerStarted","Data":"da5907575964f25954231108af720efdc5831e0c7071896530683b4f65971fb7"} Sep 10 12:32:16.302772 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:16.302703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jjtjh" Sep 10 12:32:16.327481 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:16.327437 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jjtjh" podStartSLOduration=65.775905619 podStartE2EDuration="1m7.327423696s" totalImagesPullingTime="1.551518077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:16.325563167 +0000 UTC m=+102.951760741" watchObservedRunningTime="2026-09-10 12:32:16.327423696 +0000 UTC m=+102.953621314" Sep 10 12:32:17.191726 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.191689 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:32:17.715147 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.715115 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-vhhs9"] Sep 10 12:32:17.748106 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.748085 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vhhs9"] Sep 10 12:32:17.748239 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.748164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:17.754011 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.753987 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 12:32:17.754141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.754018 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 12:32:17.754141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.754034 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dg6dt\"" Sep 10 12:32:17.868311 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.868285 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722jm\" (UniqueName: \"kubernetes.io/projected/5eb3e666-7520-40e7-b6d1-962fab87d027-kube-api-access-722jm\") pod \"downloads-85d98c998f-vhhs9\" (UID: \"5eb3e666-7520-40e7-b6d1-962fab87d027\") " pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:17.969178 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.969115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722jm\" (UniqueName: \"kubernetes.io/projected/5eb3e666-7520-40e7-b6d1-962fab87d027-kube-api-access-722jm\") pod \"downloads-85d98c998f-vhhs9\" (UID: \"5eb3e666-7520-40e7-b6d1-962fab87d027\") " pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:17.984120 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:17.984097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-722jm\" (UniqueName: \"kubernetes.io/projected/5eb3e666-7520-40e7-b6d1-962fab87d027-kube-api-access-722jm\") pod \"downloads-85d98c998f-vhhs9\" (UID: \"5eb3e666-7520-40e7-b6d1-962fab87d027\") " pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:18.057143 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:18.057118 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:18.355449 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:18.355422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vhhs9"] Sep 10 12:32:18.358679 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:18.358646 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb3e666_7520_40e7_b6d1_962fab87d027.slice/crio-d6dc596c5b629d289f30e808dd3724c97a11263befe5eee7c4f68a74cf6a848a WatchSource:0}: Error finding container d6dc596c5b629d289f30e808dd3724c97a11263befe5eee7c4f68a74cf6a848a: Status 404 returned error can't find the container with id d6dc596c5b629d289f30e808dd3724c97a11263befe5eee7c4f68a74cf6a848a Sep 10 12:32:19.284736 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.284705 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-w9prz"] Sep 10 12:32:19.310455 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.310416 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-w9prz"] Sep 10 12:32:19.311165 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.310550 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.312416 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.312392 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2wxrt"] Sep 10 12:32:19.313989 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.313968 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 12:32:19.314543 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.314278 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 12:32:19.314543 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.314336 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 12:32:19.314752 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.314732 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9689h\"" Sep 10 12:32:19.314865 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.314756 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 12:32:19.314865 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.314782 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 12:32:19.318214 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.318195 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 12:32:19.325831 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.325810 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vhhs9" event={"ID":"5eb3e666-7520-40e7-b6d1-962fab87d027","Type":"ContainerStarted","Data":"d6dc596c5b629d289f30e808dd3724c97a11263befe5eee7c4f68a74cf6a848a"} Sep 10 12:32:19.325914 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.325898 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.329737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.329720 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 12:32:19.330271 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.330068 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5j6g8\"" Sep 10 12:32:19.330271 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.330102 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 12:32:19.330271 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.330112 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 12:32:19.378918 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.378893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-tls\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.378929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c24c7cd2-ab0f-43be-935f-861896a14c1b-metrics-client-ca\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.378953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-sys\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-root\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-wtmp\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpsd6\" (UniqueName: \"kubernetes.io/projected/c24c7cd2-ab0f-43be-935f-861896a14c1b-kube-api-access-fpsd6\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379263 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-textfile\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.379748 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.379748 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.379748 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.379390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvscm\" (UniqueName: \"kubernetes.io/projected/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-api-access-fvscm\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.480286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c24c7cd2-ab0f-43be-935f-861896a14c1b-metrics-client-ca\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.480438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.480700 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.480815 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.480892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-sys\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.480953 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.481008 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480967 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-root\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.481061 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.481013 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-sys\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.481174 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.480995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-wtmp\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.481229 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.481200 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.481573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.481543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-root\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.481772 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.481527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.481840 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.481818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-wtmp\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.481943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpsd6\" (UniqueName: \"kubernetes.io/projected/c24c7cd2-ab0f-43be-935f-861896a14c1b-kube-api-access-fpsd6\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-textfile\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482272 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvscm\" (UniqueName: \"kubernetes.io/projected/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-api-access-fvscm\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-tls\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c24c7cd2-ab0f-43be-935f-861896a14c1b-metrics-client-ca\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.482029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.483116 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-textfile\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.483573 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:32:19.483468 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 10 12:32:19.484096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.483734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.486675 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.484612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.486675 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.484906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.486675 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.485268 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.486675 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:32:19.485429 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-tls podName:c24c7cd2-ab0f-43be-935f-861896a14c1b nodeName:}" failed. No retries permitted until 2026-09-10 12:32:19.985404616 +0000 UTC m=+106.611602186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-tls") pod "node-exporter-2wxrt" (UID: "c24c7cd2-ab0f-43be-935f-861896a14c1b") : secret "node-exporter-tls" not found Sep 10 12:32:19.487553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.487527 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.495380 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.495355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpsd6\" (UniqueName: \"kubernetes.io/projected/c24c7cd2-ab0f-43be-935f-861896a14c1b-kube-api-access-fpsd6\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.500467 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.500446 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvscm\" (UniqueName: \"kubernetes.io/projected/fbe55502-a5c4-40a4-88f8-16e462c1e3b4-kube-api-access-fvscm\") pod \"kube-state-metrics-74cb57df7-w9prz\" (UID: \"fbe55502-a5c4-40a4-88f8-16e462c1e3b4\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.624096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.624024 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" Sep 10 12:32:19.942808 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.942733 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-w9prz"] Sep 10 12:32:19.946180 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:19.946148 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe55502_a5c4_40a4_88f8_16e462c1e3b4.slice/crio-aa9b518522f93c00f7f9ef3414b17e5a3c28c56fec13eac09a3831fcb49a8003 WatchSource:0}: Error finding container aa9b518522f93c00f7f9ef3414b17e5a3c28c56fec13eac09a3831fcb49a8003: Status 404 returned error can't find the container with id aa9b518522f93c00f7f9ef3414b17e5a3c28c56fec13eac09a3831fcb49a8003 Sep 10 12:32:19.987501 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.987471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-tls\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:19.990053 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:19.990031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c24c7cd2-ab0f-43be-935f-861896a14c1b-node-exporter-tls\") pod \"node-exporter-2wxrt\" (UID: \"c24c7cd2-ab0f-43be-935f-861896a14c1b\") " pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:20.236383 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:20.236309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wxrt" Sep 10 12:32:20.248433 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:20.248403 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24c7cd2_ab0f_43be_935f_861896a14c1b.slice/crio-6fc92a7072591e46d46f74d38a39d9d368ec512fe01ef8995cd9d3b7efcf0426 WatchSource:0}: Error finding container 6fc92a7072591e46d46f74d38a39d9d368ec512fe01ef8995cd9d3b7efcf0426: Status 404 returned error can't find the container with id 6fc92a7072591e46d46f74d38a39d9d368ec512fe01ef8995cd9d3b7efcf0426 Sep 10 12:32:20.322102 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:20.322070 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wxrt" event={"ID":"c24c7cd2-ab0f-43be-935f-861896a14c1b","Type":"ContainerStarted","Data":"6fc92a7072591e46d46f74d38a39d9d368ec512fe01ef8995cd9d3b7efcf0426"} Sep 10 12:32:20.323603 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:20.323564 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" event={"ID":"fbe55502-a5c4-40a4-88f8-16e462c1e3b4","Type":"ContainerStarted","Data":"aa9b518522f93c00f7f9ef3414b17e5a3c28c56fec13eac09a3831fcb49a8003"} Sep 10 12:32:21.524626 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.523288 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-766c99fbbd-7fb2c"] Sep 10 12:32:21.526540 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.526514 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.531238 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.531214 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jfdrb\"" Sep 10 12:32:21.532003 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.531986 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e4foj1siee9hi\"" Sep 10 12:32:21.532433 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.532411 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 12:32:21.532845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.532657 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 12:32:21.532845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.532699 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 12:32:21.533019 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.532873 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 12:32:21.533019 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.532947 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 12:32:21.546676 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.546612 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-766c99fbbd-7fb2c"] Sep 10 12:32:21.601556 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-tls\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6254c17b-eb8d-4c15-a848-14d0f8d2e078-metrics-client-ca\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601810 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-grpc-tls\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601810 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601891 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601816 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv7bj\" (UniqueName: \"kubernetes.io/projected/6254c17b-eb8d-4c15-a848-14d0f8d2e078-kube-api-access-fv7bj\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601891 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.601891 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.601881 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.702912 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.702868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6254c17b-eb8d-4c15-a848-14d0f8d2e078-metrics-client-ca\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703010 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.702919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-grpc-tls\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703074 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703064 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703172 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv7bj\" (UniqueName: \"kubernetes.io/projected/6254c17b-eb8d-4c15-a848-14d0f8d2e078-kube-api-access-fv7bj\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703172 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703276 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703276 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-tls\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.703571 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.703450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6254c17b-eb8d-4c15-a848-14d0f8d2e078-metrics-client-ca\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.706860 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.706818 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-grpc-tls\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.706996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.706970 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.707140 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.707084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.707918 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.707894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.708025 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.708004 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-tls\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.708515 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.708479 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6254c17b-eb8d-4c15-a848-14d0f8d2e078-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.720352 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.720328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv7bj\" (UniqueName: \"kubernetes.io/projected/6254c17b-eb8d-4c15-a848-14d0f8d2e078-kube-api-access-fv7bj\") pod \"thanos-querier-766c99fbbd-7fb2c\" (UID: \"6254c17b-eb8d-4c15-a848-14d0f8d2e078\") " pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:21.839147 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:21.838884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:22.139515 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.139356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-766c99fbbd-7fb2c"] Sep 10 12:32:22.144298 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:22.144264 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6254c17b_eb8d_4c15_a848_14d0f8d2e078.slice/crio-fd47ac6357c1549b6bd6f7dc6754fbdaba8e0482c754513f9095de229defec64 WatchSource:0}: Error finding container fd47ac6357c1549b6bd6f7dc6754fbdaba8e0482c754513f9095de229defec64: Status 404 returned error can't find the container with id fd47ac6357c1549b6bd6f7dc6754fbdaba8e0482c754513f9095de229defec64 Sep 10 12:32:22.337707 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.337677 2562 generic.go:356] "Generic (PLEG): container finished" podID="c24c7cd2-ab0f-43be-935f-861896a14c1b" containerID="3b5655f8eea83751ed5368a95f59f403761089150102cb1f1cb71b904fe6af34" exitCode=0 Sep 10 12:32:22.337859 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.337752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wxrt" event={"ID":"c24c7cd2-ab0f-43be-935f-861896a14c1b","Type":"ContainerDied","Data":"3b5655f8eea83751ed5368a95f59f403761089150102cb1f1cb71b904fe6af34"} Sep 10 12:32:22.340148 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.340123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" event={"ID":"fbe55502-a5c4-40a4-88f8-16e462c1e3b4","Type":"ContainerStarted","Data":"8ba372c21f99ac2acc62547bd8d3cad417cdfd39bf4417768271bf8ed71564c0"} Sep 10 12:32:22.340254 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.340175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" event={"ID":"fbe55502-a5c4-40a4-88f8-16e462c1e3b4","Type":"ContainerStarted","Data":"4278cd30bb3cb522ce7e3d9b021301e414c967f8aaf017da122d46b6b97d32e8"} Sep 10 12:32:22.340254 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.340197 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" event={"ID":"fbe55502-a5c4-40a4-88f8-16e462c1e3b4","Type":"ContainerStarted","Data":"bdcb6c3d945b563c0f4376ca225cca624ba8d7730d00d8c2553c1bc7ec037a03"} Sep 10 12:32:22.341367 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.341337 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"fd47ac6357c1549b6bd6f7dc6754fbdaba8e0482c754513f9095de229defec64"} Sep 10 12:32:22.402124 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:22.402073 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-w9prz" podStartSLOduration=1.7066617339999999 podStartE2EDuration="3.402052321s" totalImagesPullingTime="1.695390587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:22.401828412 +0000 UTC m=+109.028025989" watchObservedRunningTime="2026-09-10 12:32:22.402052321 +0000 UTC m=+109.028249898" Sep 10 12:32:23.351628 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.351295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wxrt" event={"ID":"c24c7cd2-ab0f-43be-935f-861896a14c1b","Type":"ContainerStarted","Data":"5b2d08904b591a677412ca38729f5b3c2fe18eb614aecd169fb280dbd3a43d36"} Sep 10 12:32:23.351628 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.351363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wxrt" event={"ID":"c24c7cd2-ab0f-43be-935f-861896a14c1b","Type":"ContainerStarted","Data":"1edd23eb3a7ee268ff2fec24094b27a5032d9827f95953964eed470a42484ac8"} Sep 10 12:32:23.374489 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.374022 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2wxrt" podStartSLOduration=2.883351433 podStartE2EDuration="4.374003172s" totalImagesPullingTime="1.39453223s" totalInitContainerRuntime="96.119509ms" isStatefulPod=true podCreationTimestamp="2026-09-10 12:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:23.373017448 +0000 UTC m=+109.999215045" watchObservedRunningTime="2026-09-10 12:32:23.374003172 +0000 UTC m=+110.000200750" Sep 10 12:32:23.707569 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.707488 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cd5c8d54b-wprd5"] Sep 10 12:32:23.709902 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.709882 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.713601 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.713560 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 12:32:23.713879 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.713860 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8rttb\"" Sep 10 12:32:23.714792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.714472 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 12:32:23.714792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.714487 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 12:32:23.714792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.714521 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 12:32:23.714792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.714490 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4fr5jc38hgrqs\"" Sep 10 12:32:23.725476 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.725455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cd5c8d54b-wprd5"] Sep 10 12:32:23.822666 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjwf\" (UniqueName: \"kubernetes.io/projected/65ace7c6-1680-46b5-8cc3-40b231626375-kube-api-access-dxjwf\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.822824 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-secret-metrics-server-tls\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.822824 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/65ace7c6-1680-46b5-8cc3-40b231626375-metrics-server-audit-profiles\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.822824 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822770 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ace7c6-1680-46b5-8cc3-40b231626375-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.822824 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/65ace7c6-1680-46b5-8cc3-40b231626375-audit-log\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.823041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-client-ca-bundle\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.823041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.822873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-secret-metrics-server-client-certs\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.923739 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/65ace7c6-1680-46b5-8cc3-40b231626375-metrics-server-audit-profiles\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.923921 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ace7c6-1680-46b5-8cc3-40b231626375-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.923921 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/65ace7c6-1680-46b5-8cc3-40b231626375-audit-log\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.923921 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-client-ca-bundle\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.923921 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-secret-metrics-server-client-certs\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.924073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjwf\" (UniqueName: \"kubernetes.io/projected/65ace7c6-1680-46b5-8cc3-40b231626375-kube-api-access-dxjwf\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.924073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.923959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-secret-metrics-server-tls\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.924779 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.924557 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/65ace7c6-1680-46b5-8cc3-40b231626375-audit-log\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.925575 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.925360 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ace7c6-1680-46b5-8cc3-40b231626375-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.925575 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.925514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/65ace7c6-1680-46b5-8cc3-40b231626375-metrics-server-audit-profiles\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.927461 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.927437 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-secret-metrics-server-client-certs\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.927575 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.927512 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-secret-metrics-server-tls\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.928101 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.928053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/65ace7c6-1680-46b5-8cc3-40b231626375-client-ca-bundle\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:23.938473 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:23.938454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjwf\" (UniqueName: \"kubernetes.io/projected/65ace7c6-1680-46b5-8cc3-40b231626375-kube-api-access-dxjwf\") pod \"metrics-server-6cd5c8d54b-wprd5\" (UID: \"65ace7c6-1680-46b5-8cc3-40b231626375\") " pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:24.022622 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.022389 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:24.394552 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.394467 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595f9b5c69-ww5jt"] Sep 10 12:32:24.396880 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.396857 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.399595 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.399555 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 12:32:24.399986 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.399965 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d9q98\"" Sep 10 12:32:24.400789 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.400763 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 12:32:24.400895 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.400839 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 12:32:24.400997 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.400980 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 12:32:24.401054 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.400997 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 12:32:24.454245 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.454178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595f9b5c69-ww5jt"] Sep 10 12:32:24.529567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.529368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-oauth-serving-cert\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.529567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.529417 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-oauth-config\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.529567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.529446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-config\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.529567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.529472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwlcw\" (UniqueName: \"kubernetes.io/projected/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-kube-api-access-gwlcw\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.529567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.529552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-serving-cert\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.530002 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.529646 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-service-ca\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.630138 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.630115 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-oauth-serving-cert\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.630255 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.630150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-oauth-config\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.630255 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.630168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-config\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.630255 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.630188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwlcw\" (UniqueName: \"kubernetes.io/projected/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-kube-api-access-gwlcw\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.630255 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.630251 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-serving-cert\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.630489 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.630282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-service-ca\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.631144 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.631089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-service-ca\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.631794 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.631757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-oauth-serving-cert\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.632063 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.632044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-config\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.633150 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.633119 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-oauth-config\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.633501 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.633480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-serving-cert\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.640926 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.640900 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwlcw\" (UniqueName: \"kubernetes.io/projected/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-kube-api-access-gwlcw\") pod \"console-595f9b5c69-ww5jt\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.711124 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.711098 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:24.758015 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:24.757983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cd5c8d54b-wprd5"] Sep 10 12:32:24.763229 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:24.763187 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ace7c6_1680_46b5_8cc3_40b231626375.slice/crio-0f91866fb239ba76ad8a1a7500a537173f9e10793bb8cc4c7631f46fd9440dbb WatchSource:0}: Error finding container 0f91866fb239ba76ad8a1a7500a537173f9e10793bb8cc4c7631f46fd9440dbb: Status 404 returned error can't find the container with id 0f91866fb239ba76ad8a1a7500a537173f9e10793bb8cc4c7631f46fd9440dbb Sep 10 12:32:25.009360 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.009330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595f9b5c69-ww5jt"] Sep 10 12:32:25.013630 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:25.013577 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f9b1d8_fad4_4367_a6eb_07d7f1eda10a.slice/crio-b3f63be157e0dcbf8fe4d86c2e5b242d3346b13c5e483df97ab5cbd4f1949cc6 WatchSource:0}: Error finding container b3f63be157e0dcbf8fe4d86c2e5b242d3346b13c5e483df97ab5cbd4f1949cc6: Status 404 returned error can't find the container with id b3f63be157e0dcbf8fe4d86c2e5b242d3346b13c5e483df97ab5cbd4f1949cc6 Sep 10 12:32:25.362749 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.362660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" event={"ID":"65ace7c6-1680-46b5-8cc3-40b231626375","Type":"ContainerStarted","Data":"0f91866fb239ba76ad8a1a7500a537173f9e10793bb8cc4c7631f46fd9440dbb"} Sep 10 12:32:25.365155 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.365122 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"e055eb20f949116ac0641ccca4f81e37576deec940ac91a37e729ad1f08d8772"} Sep 10 12:32:25.365296 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.365201 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"598410e92fe716c664080509f5e21886933cfd2fb553ee0ff29b5bf3d58d4d89"} Sep 10 12:32:25.365296 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.365242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"47ccf3b76cdc56b3ba2da025d35de71d6350a8601644be260437587c578785aa"} Sep 10 12:32:25.366536 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.366514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f9b5c69-ww5jt" event={"ID":"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a","Type":"ContainerStarted","Data":"b3f63be157e0dcbf8fe4d86c2e5b242d3346b13c5e483df97ab5cbd4f1949cc6"} Sep 10 12:32:25.662055 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.661985 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:32:25.666641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.666618 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.672333 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.672315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 12:32:25.672638 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.672623 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 12:32:25.672888 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.672744 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c3uvqoac72a6h\"" Sep 10 12:32:25.673093 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.673076 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 12:32:25.673280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.673257 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 12:32:25.673367 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.673133 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rkrnb\"" Sep 10 12:32:25.673456 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.673119 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 12:32:25.673691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.673665 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 12:32:25.674711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.674528 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 12:32:25.674711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.674634 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 12:32:25.679882 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.679863 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 12:32:25.680012 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.679995 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 12:32:25.680199 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.680183 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 12:32:25.681398 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.681379 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 12:32:25.685705 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.685683 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 12:32:25.690253 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.690237 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 12:32:25.714321 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.714297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:32:25.741539 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741569 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741612 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-web-config\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741638 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-config\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741694 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-config-out\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741898 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dzrn\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-kube-api-access-8dzrn\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.741946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.742360 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.741968 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.742360 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.742009 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.742360 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.742041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.742360 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.742069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.742360 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.742127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.842906 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.842879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843035 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.842915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843035 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.842934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dzrn\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-kube-api-access-8dzrn\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843035 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.842964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843035 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.842997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843035 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-web-config\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843286 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-config\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-config-out\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.843697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.843458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.848892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.846449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.848892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.848315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.848892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.848665 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.849440 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.849405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.850633 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.850113 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.850928 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.850903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.851161 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.851139 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.851971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.851952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.852346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.852254 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.852568 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.852543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.853150 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.853063 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-config\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.855005 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.853479 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.855005 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.853835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.855005 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.853947 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.858516 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.858483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-web-config\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.858933 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.858890 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.859186 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.859152 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-config-out\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.860316 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.860293 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.860502 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.860480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.865213 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.865189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dzrn\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-kube-api-access-8dzrn\") pod \"prometheus-k8s-0\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:25.979781 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:25.979537 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:26.304769 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.304636 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:32:26.309463 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.308790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jjtjh" Sep 10 12:32:26.309721 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:26.309698 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a808e7_3aad_4328_a6e6_591783e59cb5.slice/crio-6bf0ee8a8f77a7e4f7e60e5076a91e9878beb2b724ae8060d1ed023d6b97146b WatchSource:0}: Error finding container 6bf0ee8a8f77a7e4f7e60e5076a91e9878beb2b724ae8060d1ed023d6b97146b: Status 404 returned error can't find the container with id 6bf0ee8a8f77a7e4f7e60e5076a91e9878beb2b724ae8060d1ed023d6b97146b Sep 10 12:32:26.379991 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.379840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"9a58066505c286efcff206e6a383b706f359002413f7c39ecaa23a1d845dbfe3"} Sep 10 12:32:26.379991 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.379917 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"3603a40b6c023475ddf43a341353adef737d0f5b9e49d05f23912c49dcc01783"} Sep 10 12:32:26.379991 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.379947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" event={"ID":"6254c17b-eb8d-4c15-a848-14d0f8d2e078","Type":"ContainerStarted","Data":"c8cb3a665fe8c517ab058edca759add5689da9a4317ee07fce2e237839baf986"} Sep 10 12:32:26.380881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.380527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:26.382459 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:26.382326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"6bf0ee8a8f77a7e4f7e60e5076a91e9878beb2b724ae8060d1ed023d6b97146b"} Sep 10 12:32:27.392614 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:27.392521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" event={"ID":"65ace7c6-1680-46b5-8cc3-40b231626375","Type":"ContainerStarted","Data":"a07d694dba31fd173537ace007fa6874446d26b545f526dd92c24c309c65f0f3"} Sep 10 12:32:27.420741 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:27.420675 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" podStartSLOduration=2.232006072 podStartE2EDuration="4.42065289s" totalImagesPullingTime="2.188646818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:27.415309368 +0000 UTC m=+114.041506984" watchObservedRunningTime="2026-09-10 12:32:27.42065289 +0000 UTC m=+114.046850466" Sep 10 12:32:27.422605 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:27.422214 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" podStartSLOduration=2.9281183349999997 podStartE2EDuration="6.422199087s" totalImagesPullingTime="3.494080752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:26.427016305 +0000 UTC m=+113.053213880" watchObservedRunningTime="2026-09-10 12:32:27.422199087 +0000 UTC m=+114.048396664" Sep 10 12:32:29.401235 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:29.401195 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" exitCode=0 Sep 10 12:32:29.401681 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:29.401285 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} Sep 10 12:32:29.403225 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:29.403197 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f9b5c69-ww5jt" event={"ID":"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a","Type":"ContainerStarted","Data":"9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90"} Sep 10 12:32:29.485295 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:29.485239 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595f9b5c69-ww5jt" podStartSLOduration=1.513736865 podStartE2EDuration="5.485217352s" totalImagesPullingTime="3.971480487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:29.482667888 +0000 UTC m=+116.108865464" watchObservedRunningTime="2026-09-10 12:32:29.485217352 +0000 UTC m=+116.111414927" Sep 10 12:32:32.206522 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:32.206467 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" podUID="49d8784e-6286-4b16-a293-d0ba76a79afb" containerName="registry" containerID="cri-o://a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc" gracePeriod=30 Sep 10 12:32:32.399038 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:32.399003 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-766c99fbbd-7fb2c" Sep 10 12:32:34.711919 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:34.711872 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:34.712332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:34.711973 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:34.718291 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:34.718267 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:35.436433 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:35.436401 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:32:36.198472 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.198452 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:32:36.350055 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.349979 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-trusted-ca\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350055 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350027 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-certificates\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350057 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350095 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w75sb\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-kube-api-access-w75sb\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350119 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d8784e-6286-4b16-a293-d0ba76a79afb-ca-trust-extracted\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350152 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350191 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-bound-sa-token\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350219 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-installation-pull-secrets\") pod \"49d8784e-6286-4b16-a293-d0ba76a79afb\" (UID: \"49d8784e-6286-4b16-a293-d0ba76a79afb\") " Sep 10 12:32:36.350548 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.350517 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:32:36.352282 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.352237 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:32:36.355000 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.354896 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:32:36.363068 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.363040 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d8784e-6286-4b16-a293-d0ba76a79afb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:32:36.363766 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.363666 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:32:36.363766 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.363736 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:32:36.363901 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.363779 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:32:36.365818 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.365792 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-kube-api-access-w75sb" (OuterVolumeSpecName: "kube-api-access-w75sb") pod "49d8784e-6286-4b16-a293-d0ba76a79afb" (UID: "49d8784e-6286-4b16-a293-d0ba76a79afb"). InnerVolumeSpecName "kube-api-access-w75sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:32:36.439728 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.439568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vhhs9" event={"ID":"5eb3e666-7520-40e7-b6d1-962fab87d027","Type":"ContainerStarted","Data":"92334f9825ebf3cac871192933d24eb8e6a449bf4d4a67bc665f7c9773db9244"} Sep 10 12:32:36.440260 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.440070 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:36.442001 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.441968 2562 generic.go:356] "Generic (PLEG): container finished" podID="49d8784e-6286-4b16-a293-d0ba76a79afb" containerID="a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc" exitCode=0 Sep 10 12:32:36.442104 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.442004 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" Sep 10 12:32:36.442526 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.442506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" event={"ID":"49d8784e-6286-4b16-a293-d0ba76a79afb","Type":"ContainerDied","Data":"a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc"} Sep 10 12:32:36.442712 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.442691 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55b7dccb46-4lwvm" event={"ID":"49d8784e-6286-4b16-a293-d0ba76a79afb","Type":"ContainerDied","Data":"a131e27228286bd5ae48e221224a3c378af86012c0ec2ab1365a5003456d30db"} Sep 10 12:32:36.442839 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.442774 2562 scope.go:117] "RemoveContainer" containerID="a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc" Sep 10 12:32:36.451567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451501 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-trusted-ca\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.451567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451527 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-certificates\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.451567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451542 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-image-registry-private-configuration\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.451567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451556 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w75sb\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-kube-api-access-w75sb\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.452193 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451572 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d8784e-6286-4b16-a293-d0ba76a79afb-ca-trust-extracted\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.452193 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451607 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.452193 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451620 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d8784e-6286-4b16-a293-d0ba76a79afb-bound-sa-token\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.452193 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.451633 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d8784e-6286-4b16-a293-d0ba76a79afb-installation-pull-secrets\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:32:36.455173 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.455154 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-vhhs9" Sep 10 12:32:36.457049 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.457030 2562 scope.go:117] "RemoveContainer" containerID="a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc" Sep 10 12:32:36.457518 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:32:36.457490 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc\": container with ID starting with a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc not found: ID does not exist" containerID="a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc" Sep 10 12:32:36.457639 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.457531 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc"} err="failed to get container status \"a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc\": rpc error: code = NotFound desc = could not find container \"a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc\": container with ID starting with a41c21a4001bc43222f6c6b00743a377766dc06d8aaafba1d962a338b0ffdbfc not found: ID does not exist" Sep 10 12:32:36.476956 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.476910 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-vhhs9" podStartSLOduration=1.658289825 podStartE2EDuration="19.476896163s" totalImagesPullingTime="17.818606338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:36.474719561 +0000 UTC m=+123.100917134" watchObservedRunningTime="2026-09-10 12:32:36.476896163 +0000 UTC m=+123.103093738" Sep 10 12:32:36.506739 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.506706 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55b7dccb46-4lwvm"] Sep 10 12:32:36.509334 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:36.509307 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55b7dccb46-4lwvm"] Sep 10 12:32:37.840494 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:37.840450 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d8784e-6286-4b16-a293-d0ba76a79afb" path="/var/lib/kubelet/pods/49d8784e-6286-4b16-a293-d0ba76a79afb/volumes" Sep 10 12:32:40.467956 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:40.467933 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} Sep 10 12:32:40.468306 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:40.468001 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} Sep 10 12:32:40.468306 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:40.468029 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} Sep 10 12:32:40.468306 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:40.468053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} Sep 10 12:32:41.476882 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:41.476843 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} Sep 10 12:32:41.477402 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:41.476919 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerStarted","Data":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} Sep 10 12:32:41.518470 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:41.518422 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.316503407 podStartE2EDuration="16.518406689s" totalImagesPullingTime="13.178611627s" totalInitContainerRuntime="23.291655ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:41.517307493 +0000 UTC m=+128.143505070" watchObservedRunningTime="2026-09-10 12:32:41.518406689 +0000 UTC m=+128.144604263" Sep 10 12:32:43.616799 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:43.616755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:32:43.619718 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:43.619690 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c49d065-9a7d-4b66-b420-0bd16b7dfbae-metrics-certs\") pod \"network-metrics-daemon-6s69q\" (UID: \"3c49d065-9a7d-4b66-b420-0bd16b7dfbae\") " pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:32:43.858817 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:43.858786 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bqff6\"" Sep 10 12:32:43.866481 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:43.866455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6s69q" Sep 10 12:32:44.023157 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:44.023096 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:44.023311 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:44.023174 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:32:44.216967 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:44.216893 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6s69q"] Sep 10 12:32:44.221892 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:32:44.221860 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c49d065_9a7d_4b66_b420_0bd16b7dfbae.slice/crio-2f312ba8e284f749d48d4c2cade409b3a105fdddd6de7ec8c9035eac732b8c52 WatchSource:0}: Error finding container 2f312ba8e284f749d48d4c2cade409b3a105fdddd6de7ec8c9035eac732b8c52: Status 404 returned error can't find the container with id 2f312ba8e284f749d48d4c2cade409b3a105fdddd6de7ec8c9035eac732b8c52 Sep 10 12:32:44.491642 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:44.491556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s69q" event={"ID":"3c49d065-9a7d-4b66-b420-0bd16b7dfbae","Type":"ContainerStarted","Data":"2f312ba8e284f749d48d4c2cade409b3a105fdddd6de7ec8c9035eac732b8c52"} Sep 10 12:32:45.980763 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:45.980712 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:32:46.503490 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:46.503458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s69q" event={"ID":"3c49d065-9a7d-4b66-b420-0bd16b7dfbae","Type":"ContainerStarted","Data":"0d6e56b0ad73bbb1e12d8116b0c7a512a1219b60eeb94115172e9e91207ae32b"} Sep 10 12:32:46.503721 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:46.503509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6s69q" event={"ID":"3c49d065-9a7d-4b66-b420-0bd16b7dfbae","Type":"ContainerStarted","Data":"6ac87882181798035f52f0d8e867264411f8dc2c4cda5b70df1cad2d0d43fcf5"} Sep 10 12:32:46.523365 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:46.523322 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6s69q" podStartSLOduration=131.017614567 podStartE2EDuration="2m12.523308306s" totalImagesPullingTime="1.505693739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:46.522354421 +0000 UTC m=+133.148552021" watchObservedRunningTime="2026-09-10 12:32:46.523308306 +0000 UTC m=+133.149505880" Sep 10 12:32:47.086854 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:47.086819 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595f9b5c69-ww5jt"] Sep 10 12:32:58.553114 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:58.553075 2562 generic.go:356] "Generic (PLEG): container finished" podID="88e683cb-1f68-45b1-bcd9-a85bb66c9e1c" containerID="5fd694754eaa1d1c42ce8d627f61c934f9e992cd5bf84fc0c986a3c9a4420842" exitCode=0 Sep 10 12:32:58.553493 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:58.553150 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" event={"ID":"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c","Type":"ContainerDied","Data":"5fd694754eaa1d1c42ce8d627f61c934f9e992cd5bf84fc0c986a3c9a4420842"} Sep 10 12:32:58.553493 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:58.553459 2562 scope.go:117] "RemoveContainer" containerID="5fd694754eaa1d1c42ce8d627f61c934f9e992cd5bf84fc0c986a3c9a4420842" Sep 10 12:32:59.558906 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:59.558876 2562 generic.go:356] "Generic (PLEG): container finished" podID="93b9099d-bd81-4d89-a0d8-76a95392a2a7" containerID="cd0c97cfef216d77555e744cb2fe9b11c675bd324156c8bb9affe0e1f874cc45" exitCode=0 Sep 10 12:32:59.559352 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:59.558952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" event={"ID":"93b9099d-bd81-4d89-a0d8-76a95392a2a7","Type":"ContainerDied","Data":"cd0c97cfef216d77555e744cb2fe9b11c675bd324156c8bb9affe0e1f874cc45"} Sep 10 12:32:59.559352 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:59.559341 2562 scope.go:117] "RemoveContainer" containerID="cd0c97cfef216d77555e744cb2fe9b11c675bd324156c8bb9affe0e1f874cc45" Sep 10 12:32:59.560849 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:32:59.560831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-rtqmv" event={"ID":"88e683cb-1f68-45b1-bcd9-a85bb66c9e1c","Type":"ContainerStarted","Data":"b124831e435aaa72d59841e7ed8046101dd727d78a41736bd7463d1bb2166ff3"} Sep 10 12:33:00.566380 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:00.566348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l2dc2" event={"ID":"93b9099d-bd81-4d89-a0d8-76a95392a2a7","Type":"ContainerStarted","Data":"dabfc23c5e5802331cd35e3f7ef0e3e4bafb5353660d2f2fe95c3c049a8a6703"} Sep 10 12:33:04.028935 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:04.028898 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:33:04.033042 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:04.033014 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cd5c8d54b-wprd5" Sep 10 12:33:12.107218 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.107185 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-595f9b5c69-ww5jt" podUID="38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" containerName="console" containerID="cri-o://9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90" gracePeriod=15 Sep 10 12:33:12.379814 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.379781 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:33:12.444031 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444010 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-oauth-config\") pod \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " Sep 10 12:33:12.444161 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444081 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-serving-cert\") pod \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " Sep 10 12:33:12.444161 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444117 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-oauth-serving-cert\") pod \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " Sep 10 12:33:12.444248 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444160 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwlcw\" (UniqueName: \"kubernetes.io/projected/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-kube-api-access-gwlcw\") pod \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " Sep 10 12:33:12.444248 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444197 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-service-ca\") pod \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " Sep 10 12:33:12.444248 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444226 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-config\") pod \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\" (UID: \"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a\") " Sep 10 12:33:12.444565 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444533 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" (UID: "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:12.444743 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444719 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-service-ca" (OuterVolumeSpecName: "service-ca") pod "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" (UID: "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:12.445041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.444984 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-config" (OuterVolumeSpecName: "console-config") pod "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" (UID: "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:12.446483 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.446457 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" (UID: "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:12.446965 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.446945 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" (UID: "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:12.446965 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.446951 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-kube-api-access-gwlcw" (OuterVolumeSpecName: "kube-api-access-gwlcw") pod "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" (UID: "38f9b1d8-fad4-4367-a6eb-07d7f1eda10a"). InnerVolumeSpecName "kube-api-access-gwlcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:33:12.545572 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.545546 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-oauth-config\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:12.545572 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.545569 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-serving-cert\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:12.545711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.545593 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-oauth-serving-cert\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:12.545711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.545602 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwlcw\" (UniqueName: \"kubernetes.io/projected/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-kube-api-access-gwlcw\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:12.545711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.545613 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-service-ca\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:12.545711 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.545621 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a-console-config\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:12.616456 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.616396 2562 generic.go:356] "Generic (PLEG): container finished" podID="38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" containerID="9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90" exitCode=0 Sep 10 12:33:12.616456 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.616438 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595f9b5c69-ww5jt" Sep 10 12:33:12.616636 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.616468 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f9b5c69-ww5jt" event={"ID":"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a","Type":"ContainerDied","Data":"9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90"} Sep 10 12:33:12.616636 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.616520 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595f9b5c69-ww5jt" event={"ID":"38f9b1d8-fad4-4367-a6eb-07d7f1eda10a","Type":"ContainerDied","Data":"b3f63be157e0dcbf8fe4d86c2e5b242d3346b13c5e483df97ab5cbd4f1949cc6"} Sep 10 12:33:12.616636 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.616549 2562 scope.go:117] "RemoveContainer" containerID="9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90" Sep 10 12:33:12.635280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.635264 2562 scope.go:117] "RemoveContainer" containerID="9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90" Sep 10 12:33:12.635551 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:12.635530 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90\": container with ID starting with 9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90 not found: ID does not exist" containerID="9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90" Sep 10 12:33:12.635607 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.635560 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90"} err="failed to get container status \"9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90\": rpc error: code = NotFound desc = could not find container \"9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90\": container with ID starting with 9ef2c77fe426371ddae64c8ac0a5e5f40f2ec522faf7d775a78cf78568826d90 not found: ID does not exist" Sep 10 12:33:12.642663 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.642644 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595f9b5c69-ww5jt"] Sep 10 12:33:12.646802 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:12.646783 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-595f9b5c69-ww5jt"] Sep 10 12:33:13.837674 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:13.837643 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" path="/var/lib/kubelet/pods/38f9b1d8-fad4-4367-a6eb-07d7f1eda10a/volumes" Sep 10 12:33:25.980443 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:25.980359 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:26.060969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:26.060945 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:26.692041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:26.692016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:44.167043 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167003 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:33:44.167566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167320 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="prometheus" containerID="cri-o://1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" gracePeriod=600 Sep 10 12:33:44.167566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167352 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy" containerID="cri-o://2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" gracePeriod=600 Sep 10 12:33:44.167566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167382 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="thanos-sidecar" containerID="cri-o://d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" gracePeriod=600 Sep 10 12:33:44.167566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167422 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-web" containerID="cri-o://10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" gracePeriod=600 Sep 10 12:33:44.167566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167382 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" gracePeriod=600 Sep 10 12:33:44.167566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.167413 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="config-reloader" containerID="cri-o://91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" gracePeriod=600 Sep 10 12:33:44.404453 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.404430 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:44.595901 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.595873 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-db\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.595918 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-metrics-client-ca\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.595939 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-tls\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.595954 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-serving-certs-ca-bundle\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.595983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-config-out\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596096 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596026 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-kubelet-serving-ca-bundle\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596358 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596251 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-1\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596358 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596293 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-2\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596358 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596326 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-kube-rbac-proxy\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596368 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.596510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596388 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-tls-assets\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596442 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-trusted-ca-bundle\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596445 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.596510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dzrn\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-kube-api-access-8dzrn\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596519 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596548 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-grpc-tls\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596610 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-metrics-client-certs\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596641 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-thanos-prometheus-http-client-file\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596683 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-0\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596711 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-config\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596737 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-web-config\") pod \"77a808e7-3aad-4328-a6e6-591783e59cb5\" (UID: \"77a808e7-3aad-4328-a6e6-591783e59cb5\") " Sep 10 12:33:44.596825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.596809 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597028 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597054 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597158 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-metrics-client-ca\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597232 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597253 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597267 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597284 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597285 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:33:44.597354 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.597323 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.599881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.599768 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-config-out" (OuterVolumeSpecName: "config-out") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:33:44.599881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.599812 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.599881 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.599834 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.600169 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.600147 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.600607 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.600564 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.600951 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.600921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:33:44.601421 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.601390 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.601538 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.601497 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.601954 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.601928 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:33:44.602067 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.602045 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-kube-api-access-8dzrn" (OuterVolumeSpecName: "kube-api-access-8dzrn") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "kube-api-access-8dzrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:33:44.602253 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.602232 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-config" (OuterVolumeSpecName: "config") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.602658 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.602641 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.611478 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.611459 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-web-config" (OuterVolumeSpecName: "web-config") pod "77a808e7-3aad-4328-a6e6-591783e59cb5" (UID: "77a808e7-3aad-4328-a6e6-591783e59cb5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:33:44.698154 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698124 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-kube-rbac-proxy\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698154 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698151 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698166 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-tls-assets\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698178 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dzrn\" (UniqueName: \"kubernetes.io/projected/77a808e7-3aad-4328-a6e6-591783e59cb5-kube-api-access-8dzrn\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698192 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698203 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-grpc-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698216 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-metrics-client-certs\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698227 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698241 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698254 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-config\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698267 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-web-config\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698295 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-prometheus-k8s-db\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698311 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/77a808e7-3aad-4328-a6e6-591783e59cb5-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698327 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77a808e7-3aad-4328-a6e6-591783e59cb5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.698738 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.698340 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/77a808e7-3aad-4328-a6e6-591783e59cb5-config-out\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:33:44.749228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749201 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" exitCode=0 Sep 10 12:33:44.749228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749224 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" exitCode=0 Sep 10 12:33:44.749228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749231 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" exitCode=0 Sep 10 12:33:44.749420 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749237 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" exitCode=0 Sep 10 12:33:44.749420 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749242 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" exitCode=0 Sep 10 12:33:44.749420 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749247 2562 generic.go:356] "Generic (PLEG): container finished" podID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" exitCode=0 Sep 10 12:33:44.749420 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749337 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:44.749420 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} Sep 10 12:33:44.749685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749467 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} Sep 10 12:33:44.749685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749515 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} Sep 10 12:33:44.749685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749530 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.749685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} Sep 10 12:33:44.749685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} Sep 10 12:33:44.749685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} Sep 10 12:33:44.749917 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.749696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"77a808e7-3aad-4328-a6e6-591783e59cb5","Type":"ContainerDied","Data":"6bf0ee8a8f77a7e4f7e60e5076a91e9878beb2b724ae8060d1ed023d6b97146b"} Sep 10 12:33:44.758986 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.758974 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.765916 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.765901 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.772441 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.772419 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.778314 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.778288 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:33:44.779445 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.779435 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.787782 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.787724 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.793305 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.793282 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:33:44.796046 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.796031 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.802626 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.802611 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.802890 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.802872 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.802936 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.802913 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} err="failed to get container status \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" Sep 10 12:33:44.802936 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.802931 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.803150 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.803136 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.803185 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803153 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} err="failed to get container status \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" Sep 10 12:33:44.803185 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803165 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.803360 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.803346 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.803402 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803362 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} err="failed to get container status \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" Sep 10 12:33:44.803402 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803373 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.803563 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.803550 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.803675 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803565 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} err="failed to get container status \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" Sep 10 12:33:44.803675 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803575 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.803872 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.803855 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.803918 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803878 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} err="failed to get container status \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" Sep 10 12:33:44.803918 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.803891 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.804119 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.804102 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.804159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804124 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} err="failed to get container status \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" Sep 10 12:33:44.804159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804142 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.804365 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:33:44.804338 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.804409 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804368 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} err="failed to get container status \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" Sep 10 12:33:44.804409 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804379 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.804617 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804596 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} err="failed to get container status \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" Sep 10 12:33:44.804668 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804618 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.804825 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804810 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} err="failed to get container status \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" Sep 10 12:33:44.804862 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804826 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.805013 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.804993 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} err="failed to get container status \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" Sep 10 12:33:44.805052 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805014 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.805203 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805188 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} err="failed to get container status \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" Sep 10 12:33:44.805260 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805203 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.805409 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805391 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} err="failed to get container status \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" Sep 10 12:33:44.805471 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805412 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.805634 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805616 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} err="failed to get container status \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" Sep 10 12:33:44.805689 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805634 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.805838 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805819 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} err="failed to get container status \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" Sep 10 12:33:44.805907 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.805839 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.806117 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806102 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} err="failed to get container status \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" Sep 10 12:33:44.806164 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806117 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.806345 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806326 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} err="failed to get container status \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" Sep 10 12:33:44.806415 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806346 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.806601 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806564 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} err="failed to get container status \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" Sep 10 12:33:44.806650 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806603 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.806816 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806799 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} err="failed to get container status \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" Sep 10 12:33:44.806856 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.806818 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.807026 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807010 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} err="failed to get container status \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" Sep 10 12:33:44.807076 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807026 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.807259 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807242 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} err="failed to get container status \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" Sep 10 12:33:44.807301 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807259 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.807471 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807456 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} err="failed to get container status \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" Sep 10 12:33:44.807515 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807471 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.807721 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807705 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} err="failed to get container status \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" Sep 10 12:33:44.807721 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807720 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.807934 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807916 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} err="failed to get container status \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" Sep 10 12:33:44.807971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.807936 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.808165 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808148 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} err="failed to get container status \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" Sep 10 12:33:44.808233 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808167 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.808388 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808371 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} err="failed to get container status \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" Sep 10 12:33:44.808429 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808389 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.808655 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808632 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} err="failed to get container status \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" Sep 10 12:33:44.808655 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808648 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.808862 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808846 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} err="failed to get container status \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" Sep 10 12:33:44.808907 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.808863 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.809050 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809035 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} err="failed to get container status \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" Sep 10 12:33:44.809092 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809051 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.809238 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809220 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} err="failed to get container status \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" Sep 10 12:33:44.809309 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809240 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.809505 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809487 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} err="failed to get container status \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" Sep 10 12:33:44.809560 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809506 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.809758 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809741 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} err="failed to get container status \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" Sep 10 12:33:44.809798 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809758 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.809982 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809966 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} err="failed to get container status \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" Sep 10 12:33:44.810030 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.809982 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.810208 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810190 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} err="failed to get container status \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" Sep 10 12:33:44.810260 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810210 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.810415 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810391 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} err="failed to get container status \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" Sep 10 12:33:44.810415 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810408 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.810651 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810628 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} err="failed to get container status \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" Sep 10 12:33:44.810714 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810653 2562 scope.go:117] "RemoveContainer" containerID="da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d" Sep 10 12:33:44.810892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810875 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d"} err="failed to get container status \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": rpc error: code = NotFound desc = could not find container \"da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d\": container with ID starting with da21a2cad7899ec7186aebba00a19b01f1a0c2563184e18fd474f84287362f9d not found: ID does not exist" Sep 10 12:33:44.810941 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.810892 2562 scope.go:117] "RemoveContainer" containerID="2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b" Sep 10 12:33:44.811135 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811118 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b"} err="failed to get container status \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": rpc error: code = NotFound desc = could not find container \"2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b\": container with ID starting with 2b3de5116b4083aa911e3bc813f8bcb2438f1c9e51e14ebda402ddedb8fee96b not found: ID does not exist" Sep 10 12:33:44.811186 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811136 2562 scope.go:117] "RemoveContainer" containerID="10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43" Sep 10 12:33:44.811368 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811344 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43"} err="failed to get container status \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": rpc error: code = NotFound desc = could not find container \"10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43\": container with ID starting with 10e4d7d7efb11765bc27aa8965b24435a0708e03d0c3a98b64f57feb4f29ee43 not found: ID does not exist" Sep 10 12:33:44.811420 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811369 2562 scope.go:117] "RemoveContainer" containerID="d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8" Sep 10 12:33:44.811575 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811556 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8"} err="failed to get container status \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": rpc error: code = NotFound desc = could not find container \"d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8\": container with ID starting with d1526efe9e3471ce6b54c96cab2a3660c9e4613c72e2fb9372083528fb99f5f8 not found: ID does not exist" Sep 10 12:33:44.811639 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811576 2562 scope.go:117] "RemoveContainer" containerID="91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7" Sep 10 12:33:44.811843 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811825 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7"} err="failed to get container status \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": rpc error: code = NotFound desc = could not find container \"91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7\": container with ID starting with 91be6c5d31ec434583fe9224f0158bd191a864b0b324add569ae58dcf25900d7 not found: ID does not exist" Sep 10 12:33:44.811888 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.811844 2562 scope.go:117] "RemoveContainer" containerID="1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295" Sep 10 12:33:44.812118 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.812100 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295"} err="failed to get container status \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": rpc error: code = NotFound desc = could not find container \"1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295\": container with ID starting with 1399cc69f25c400f2d0d9bcb793390abb6014c96c7e8cad47786e55027d08295 not found: ID does not exist" Sep 10 12:33:44.812118 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.812118 2562 scope.go:117] "RemoveContainer" containerID="667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027" Sep 10 12:33:44.812372 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.812354 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027"} err="failed to get container status \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": rpc error: code = NotFound desc = could not find container \"667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027\": container with ID starting with 667dd5cc9241128a2c77f548294228bf6d60c4b1d235326a5b133bcdf2b85027 not found: ID does not exist" Sep 10 12:33:44.827882 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.827863 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:33:44.828137 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828126 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="config-reloader" Sep 10 12:33:44.828177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828139 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="config-reloader" Sep 10 12:33:44.828177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828149 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy" Sep 10 12:33:44.828177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828155 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy" Sep 10 12:33:44.828177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828166 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="thanos-sidecar" Sep 10 12:33:44.828177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828172 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="thanos-sidecar" Sep 10 12:33:44.828177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828177 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-thanos" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828182 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-thanos" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828191 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="init-config-reloader" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828196 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="init-config-reloader" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828202 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="prometheus" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828207 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="prometheus" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828213 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d8784e-6286-4b16-a293-d0ba76a79afb" containerName="registry" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828218 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d8784e-6286-4b16-a293-d0ba76a79afb" containerName="registry" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828225 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" containerName="console" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828229 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" containerName="console" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828237 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-web" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828242 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-web" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828286 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d8784e-6286-4b16-a293-d0ba76a79afb" containerName="registry" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828295 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828302 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="thanos-sidecar" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828307 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f9b1d8-fad4-4367-a6eb-07d7f1eda10a" containerName="console" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828311 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="prometheus" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828315 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="config-reloader" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828321 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-web" Sep 10 12:33:44.828346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.828325 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" containerName="kube-rbac-proxy-thanos" Sep 10 12:33:44.830521 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.830506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:44.851500 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.851454 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:33:44.852139 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852123 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 10 12:33:44.852290 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852266 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 10 12:33:44.852428 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852365 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 10 12:33:44.852557 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852535 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c3uvqoac72a6h\"" Sep 10 12:33:44.852698 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852535 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 10 12:33:44.852698 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852565 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 10 12:33:44.852843 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852735 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 10 12:33:44.852843 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852795 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 10 12:33:44.852843 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852809 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 10 12:33:44.852988 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.852952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 10 12:33:44.853256 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.853206 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 10 12:33:44.853394 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.853208 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 10 12:33:44.855776 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.855759 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rkrnb\"" Sep 10 12:33:44.856664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.856647 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 10 12:33:44.857808 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.857787 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 10 12:33:44.864676 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:44.864657 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 10 12:33:45.000631 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000600 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000771 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000634 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000771 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000771 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-web-config\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000771 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000764 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000892 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-config\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.000969 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.000984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001002 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zpv6\" (UniqueName: \"kubernetes.io/projected/15f309fb-6ae8-423b-bd84-20a30985db24-kube-api-access-5zpv6\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001033 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15f309fb-6ae8-423b-bd84-20a30985db24-config-out\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15f309fb-6ae8-423b-bd84-20a30985db24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.001267 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.001201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.101925 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.101848 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.101925 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.101894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-config\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.101925 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.101926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.101950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.101974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zpv6\" (UniqueName: \"kubernetes.io/projected/15f309fb-6ae8-423b-bd84-20a30985db24-kube-api-access-5zpv6\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15f309fb-6ae8-423b-bd84-20a30985db24-config-out\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15f309fb-6ae8-423b-bd84-20a30985db24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102159 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102294 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-web-config\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102395 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.102547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.103071 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.102994 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.103213 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.103804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.105170 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/15f309fb-6ae8-423b-bd84-20a30985db24-config-out\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.105326 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.105373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-config\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.105449 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.105897 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.105500 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.106115 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.106092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.106473 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.106174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.106473 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.106269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.106669 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.106617 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.106908 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.106886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.107469 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.107451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/15f309fb-6ae8-423b-bd84-20a30985db24-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.107892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.107873 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/15f309fb-6ae8-423b-bd84-20a30985db24-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.108440 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.108417 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.108483 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.108462 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-web-config\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.108811 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.108790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/15f309fb-6ae8-423b-bd84-20a30985db24-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.110698 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.110682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zpv6\" (UniqueName: \"kubernetes.io/projected/15f309fb-6ae8-423b-bd84-20a30985db24-kube-api-access-5zpv6\") pod \"prometheus-k8s-0\" (UID: \"15f309fb-6ae8-423b-bd84-20a30985db24\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.139041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.139021 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:33:45.426570 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.426544 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 10 12:33:45.428945 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:33:45.428922 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f309fb_6ae8_423b_bd84_20a30985db24.slice/crio-2f88bce7e175f3918d47860f9af8bc2a37d7cd736b2168e00a068775d81365d2 WatchSource:0}: Error finding container 2f88bce7e175f3918d47860f9af8bc2a37d7cd736b2168e00a068775d81365d2: Status 404 returned error can't find the container with id 2f88bce7e175f3918d47860f9af8bc2a37d7cd736b2168e00a068775d81365d2 Sep 10 12:33:45.755968 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.755934 2562 generic.go:356] "Generic (PLEG): container finished" podID="15f309fb-6ae8-423b-bd84-20a30985db24" containerID="16e2aaeedcfbb7956f9acbb47ef2b9dfddabe2ebea4a1963339f10918799d8f2" exitCode=0 Sep 10 12:33:45.756095 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.756024 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerDied","Data":"16e2aaeedcfbb7956f9acbb47ef2b9dfddabe2ebea4a1963339f10918799d8f2"} Sep 10 12:33:45.756131 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.756109 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"2f88bce7e175f3918d47860f9af8bc2a37d7cd736b2168e00a068775d81365d2"} Sep 10 12:33:45.840128 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:45.840093 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77a808e7-3aad-4328-a6e6-591783e59cb5" path="/var/lib/kubelet/pods/77a808e7-3aad-4328-a6e6-591783e59cb5/volumes" Sep 10 12:33:46.766444 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.766406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"eae33c50bc884d3477547af8d93536422ffd2de6402b8c860548dea1522f3be9"} Sep 10 12:33:46.766832 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.766486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"bb3c2f4fc6121e3e77d23690c52c54ea716febd91dfd6d28114e104397d0c1a0"} Sep 10 12:33:46.766832 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.766525 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"5a70165fb07dc9a5e182f06432766854de1c39393bca82a37ac0f0e5397ab2d8"} Sep 10 12:33:46.766832 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.766553 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"640c9cb90539f6662a48ae627c1cbb36f3c540796215e2f99095ecdefa0299a6"} Sep 10 12:33:46.766832 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.766601 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"d93a2dd01cd33ce3c3cf2da389a8e1fd20d2d09721295930deb42b8b08d4c486"} Sep 10 12:33:46.766832 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.766631 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"15f309fb-6ae8-423b-bd84-20a30985db24","Type":"ContainerStarted","Data":"96278aa5581a62113c78a5b6573889893ece20b6574344ad3b8ba0ca9d871c72"} Sep 10 12:33:46.804207 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:46.804158 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7832577130000002 podStartE2EDuration="2.804138371s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.880658ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:33:46.80305835 +0000 UTC m=+193.429255970" watchObservedRunningTime="2026-09-10 12:33:46.804138371 +0000 UTC m=+193.430335948" Sep 10 12:33:50.139899 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:33:50.139856 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:34:45.140262 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:34:45.140228 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:34:45.156400 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:34:45.156378 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:34:46.013388 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:34:46.013357 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 10 12:35:33.729466 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:35:33.729444 2562 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 12:37:41.917254 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.917221 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg"] Sep 10 12:37:41.920182 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.920163 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:41.924089 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.924066 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 10 12:37:41.924199 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.924132 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 10 12:37:41.924791 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.924772 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 10 12:37:41.924988 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.924971 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 10 12:37:41.925168 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.925154 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-c2fx2\"" Sep 10 12:37:41.925230 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.925170 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 10 12:37:41.928300 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.928278 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 10 12:37:41.928426 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.928373 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 10 12:37:41.948794 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:41.948773 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg"] Sep 10 12:37:42.016157 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.016133 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-offline-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.016274 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.016159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fw82\" (UniqueName: \"kubernetes.io/projected/39f344d8-6329-495c-ab26-6e2b58e3f401-kube-api-access-2fw82\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.016274 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.016182 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-ui-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.016274 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.016251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39f344d8-6329-495c-ab26-6e2b58e3f401-feast-data\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.016429 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.016319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.016429 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.016350 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-registry-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117196 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-registry-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-offline-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fw82\" (UniqueName: \"kubernetes.io/projected/39f344d8-6329-495c-ab26-6e2b58e3f401-kube-api-access-2fw82\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-ui-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39f344d8-6329-495c-ab26-6e2b58e3f401-feast-data\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.117690 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:37:42.117426 2562 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 10 12:37:42.117690 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:37:42.117501 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls podName:39f344d8-6329-495c-ab26-6e2b58e3f401 nodeName:}" failed. No retries permitted until 2026-09-10 12:37:42.617480296 +0000 UTC m=+429.243677852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls") pod "feast-simple-feast-setup-789ddffb9f-l2ggg" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401") : secret "feast-simple-feast-setup-online-tls" not found Sep 10 12:37:42.117790 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.117736 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39f344d8-6329-495c-ab26-6e2b58e3f401-feast-data\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.119747 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.119723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-ui-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.119848 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.119820 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-registry-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.119888 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.119861 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-offline-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.130417 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.130394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fw82\" (UniqueName: \"kubernetes.io/projected/39f344d8-6329-495c-ab26-6e2b58e3f401-kube-api-access-2fw82\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.620029 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.619997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.622553 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.622531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-l2ggg\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:42.845226 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:42.845185 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:37:43.124298 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:43.124271 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg"] Sep 10 12:37:43.125596 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:37:43.125545 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39f344d8_6329_495c_ab26_6e2b58e3f401.slice/crio-cca02e29fb7078333c2a338040b463c4c930fecc10d5d828e3352d086587789d WatchSource:0}: Error finding container cca02e29fb7078333c2a338040b463c4c930fecc10d5d828e3352d086587789d: Status 404 returned error can't find the container with id cca02e29fb7078333c2a338040b463c4c930fecc10d5d828e3352d086587789d Sep 10 12:37:43.127565 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:43.127550 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:37:43.690905 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:37:43.690871 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"cca02e29fb7078333c2a338040b463c4c930fecc10d5d828e3352d086587789d"} Sep 10 12:38:11.811477 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:11.811433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"6609ffeb7cd38077217081648c9a182afd4fe39ef09fdaa01364993248344de6"} Sep 10 12:38:15.831970 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:15.831936 2562 generic.go:356] "Generic (PLEG): container finished" podID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerID="6609ffeb7cd38077217081648c9a182afd4fe39ef09fdaa01364993248344de6" exitCode=0 Sep 10 12:38:15.832339 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:15.832009 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"6609ffeb7cd38077217081648c9a182afd4fe39ef09fdaa01364993248344de6"} Sep 10 12:38:16.838707 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:16.838676 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"33e1ed7496d2480e342f7dfb2650a4f967b321086ebf01a85a1bfd1b432f10da"} Sep 10 12:38:21.858197 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:21.858163 2562 generic.go:356] "Generic (PLEG): container finished" podID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerID="33e1ed7496d2480e342f7dfb2650a4f967b321086ebf01a85a1bfd1b432f10da" exitCode=0 Sep 10 12:38:21.858570 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:21.858240 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"33e1ed7496d2480e342f7dfb2650a4f967b321086ebf01a85a1bfd1b432f10da"} Sep 10 12:38:22.868071 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:22.868038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"92ad1f45d8d0e037bd2bca36bebbf033b790cbe5f25191a9a051990218ff8372"} Sep 10 12:38:22.868534 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:22.868094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"611828f9e7ad9626e73c62e0c0fa5af440a60c715eb79f33db7c0965a1c729d5"} Sep 10 12:38:22.868534 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:22.868116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"9c366d1d0a90a18b66c8043dfb6939d09526159277d96f072288999769856df9"} Sep 10 12:38:22.868534 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:22.868136 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerStarted","Data":"bf625417e406e4535ce23e91c260d6867fd0437f467d4aa3e34f1562567fd3e2"} Sep 10 12:38:22.897724 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:22.897665 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podStartSLOduration=14.209694015 podStartE2EDuration="41.89764294s" totalImagesPullingTime="27.687948925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:22.895803933 +0000 UTC m=+469.522001523" watchObservedRunningTime="2026-09-10 12:38:22.89764294 +0000 UTC m=+469.523840518" Sep 10 12:38:24.846369 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.846322 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:24.846861 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.846394 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:24.846861 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.846415 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:24.846861 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.846432 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:24.848144 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.848090 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 10 12:38:24.848381 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.848107 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 10 12:38:24.848508 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.848150 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 10 12:38:24.848508 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:24.848166 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 10 12:38:27.845938 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:27.845732 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 10 12:38:27.845938 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:27.845788 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 10 12:38:27.845938 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:27.845732 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.23:6571: connect: connection refused" Sep 10 12:38:27.845938 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:27.845732 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 10 12:38:30.846700 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.846666 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.847080 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.846809 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.847366 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.847345 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.847496 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.847385 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.847496 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.847488 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.847782 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.847754 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.847851 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.847808 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.852251 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.852230 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.912873 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.912841 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.913013 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.912880 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.913252 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.913234 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:38:30.915909 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:38:30.915895 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:39:34.161037 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.160951 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg"] Sep 10 12:39:34.161560 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.161277 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="registry" containerID="cri-o://bf625417e406e4535ce23e91c260d6867fd0437f467d4aa3e34f1562567fd3e2" gracePeriod=30 Sep 10 12:39:34.161560 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.161351 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" containerID="cri-o://9c366d1d0a90a18b66c8043dfb6939d09526159277d96f072288999769856df9" gracePeriod=30 Sep 10 12:39:34.161560 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.161356 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="offline" containerID="cri-o://611828f9e7ad9626e73c62e0c0fa5af440a60c715eb79f33db7c0965a1c729d5" gracePeriod=30 Sep 10 12:39:34.161560 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.161390 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" containerID="cri-o://92ad1f45d8d0e037bd2bca36bebbf033b790cbe5f25191a9a051990218ff8372" gracePeriod=30 Sep 10 12:39:34.629786 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.629745 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4"] Sep 10 12:39:34.633223 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.633204 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.639142 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.639118 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-8lqsj\"" Sep 10 12:39:34.657412 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.657380 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4"] Sep 10 12:39:34.712693 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.712656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-registry-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.712859 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.712705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.712859 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.712727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-ui-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.712859 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.712784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-offline-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.712859 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.712828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-feast-data\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.712859 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.712850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshl6\" (UniqueName: \"kubernetes.io/projected/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-kube-api-access-cshl6\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814025 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.813997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814185 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.814038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-ui-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814185 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.814163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-offline-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814293 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.814226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-feast-data\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814293 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.814246 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshl6\" (UniqueName: \"kubernetes.io/projected/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-kube-api-access-cshl6\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.814308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-registry-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.814640 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.814607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-feast-data\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.816678 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.816657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-online-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.816909 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.816883 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-registry-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.817026 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.817009 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-offline-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.817246 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.817228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-ui-tls\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.822907 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.822889 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshl6\" (UniqueName: \"kubernetes.io/projected/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-kube-api-access-cshl6\") pod \"feast-simple-feast-setup-789ddffb9f-w7lq4\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:34.943754 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:34.943719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:35.167107 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:35.167036 2562 generic.go:356] "Generic (PLEG): container finished" podID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerID="92ad1f45d8d0e037bd2bca36bebbf033b790cbe5f25191a9a051990218ff8372" exitCode=0 Sep 10 12:39:35.167107 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:35.167057 2562 generic.go:356] "Generic (PLEG): container finished" podID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerID="9c366d1d0a90a18b66c8043dfb6939d09526159277d96f072288999769856df9" exitCode=0 Sep 10 12:39:35.167107 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:35.167081 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"92ad1f45d8d0e037bd2bca36bebbf033b790cbe5f25191a9a051990218ff8372"} Sep 10 12:39:35.167489 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:35.167113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"9c366d1d0a90a18b66c8043dfb6939d09526159277d96f072288999769856df9"} Sep 10 12:39:35.220089 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:35.219943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4"] Sep 10 12:39:35.222495 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:39:35.222467 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bf6ed68_fd15_4b9c_b319_3e45709e8be5.slice/crio-0edf0e259eb22edc4d9ac99b59c19648b9b0785be167cbb7e666fb27270f6561 WatchSource:0}: Error finding container 0edf0e259eb22edc4d9ac99b59c19648b9b0785be167cbb7e666fb27270f6561: Status 404 returned error can't find the container with id 0edf0e259eb22edc4d9ac99b59c19648b9b0785be167cbb7e666fb27270f6561 Sep 10 12:39:36.172545 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:36.172512 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"8773053f582ebd8a4a2c0fd016a0f79dfc67cb685c36f17e7e46288746b1c242"} Sep 10 12:39:36.172928 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:36.172560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"0edf0e259eb22edc4d9ac99b59c19648b9b0785be167cbb7e666fb27270f6561"} Sep 10 12:39:40.190929 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:40.190897 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerID="8773053f582ebd8a4a2c0fd016a0f79dfc67cb685c36f17e7e46288746b1c242" exitCode=0 Sep 10 12:39:40.191224 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:40.190970 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"8773053f582ebd8a4a2c0fd016a0f79dfc67cb685c36f17e7e46288746b1c242"} Sep 10 12:39:40.848195 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:40.848149 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 10 12:39:40.913872 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:40.913820 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 10 12:39:41.197099 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:41.197018 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"58c87cd9e5f692c473c85737742363dfee25fdc88b5930ac8975884f962f89de"} Sep 10 12:39:46.219914 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:46.219829 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerID="58c87cd9e5f692c473c85737742363dfee25fdc88b5930ac8975884f962f89de" exitCode=0 Sep 10 12:39:46.219914 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:46.219868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"58c87cd9e5f692c473c85737742363dfee25fdc88b5930ac8975884f962f89de"} Sep 10 12:39:47.231156 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:47.231114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"07b1793818cbaa95aeae2ae6ec4b8c41c24dc70e93e0e6b4c4cd4c092bb19bc9"} Sep 10 12:39:47.231620 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:47.231175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"9e10f54b5bfb278de6bf8d582fd9e399f6b18ff157f488e0576f9d5511a64233"} Sep 10 12:39:47.231620 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:47.231195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"6d746bafde52e7625e9bc03780febbef9ede55b9eec994bf481bdcfb96fcf936"} Sep 10 12:39:47.231620 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:47.231215 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerStarted","Data":"dc6636011719d5d6128df993d03de68f0bdf1454b8f2c1bf31e456cd9075ecc9"} Sep 10 12:39:47.260664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:47.260603 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podStartSLOduration=13.260557282 podStartE2EDuration="13.260557282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:47.258692103 +0000 UTC m=+553.884889681" watchObservedRunningTime="2026-09-10 12:39:47.260557282 +0000 UTC m=+553.886754858" Sep 10 12:39:49.944858 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.944819 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:49.945320 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.944882 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:49.945320 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.944902 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:49.945320 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.944919 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:49.947075 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.947037 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 10 12:39:49.947075 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.947059 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 10 12:39:49.947271 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.947081 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 10 12:39:49.947271 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:49.947138 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 10 12:39:50.848301 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:50.848196 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 10 12:39:50.913984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:50.913897 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 10 12:39:52.944522 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:52.944472 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.24:6571: connect: connection refused" Sep 10 12:39:52.945029 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:52.944472 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 10 12:39:52.945029 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:52.944474 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.24:8816: connect: connection refused" Sep 10 12:39:52.945213 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:52.945174 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 10 12:39:54.944097 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:54.944063 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:54.944450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:54.944116 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:54.944450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:54.944130 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:54.944450 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:54.944141 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.945896 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.945865 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.946372 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.946019 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.946532 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.946503 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.946884 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.946858 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.947020 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.946999 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.947116 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.947102 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:55.951413 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:55.951393 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:39:56.283805 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:39:56.283782 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:40:00.848162 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:00.848115 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 10 12:40:00.848552 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:00.848291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:40:00.913395 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:00.913364 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 10 12:40:00.913555 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:00.913461 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:40:04.317063 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.317035 2562 generic.go:356] "Generic (PLEG): container finished" podID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerID="611828f9e7ad9626e73c62e0c0fa5af440a60c715eb79f33db7c0965a1c729d5" exitCode=137 Sep 10 12:40:04.317063 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.317060 2562 generic.go:356] "Generic (PLEG): container finished" podID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerID="bf625417e406e4535ce23e91c260d6867fd0437f467d4aa3e34f1562567fd3e2" exitCode=137 Sep 10 12:40:04.317373 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.317107 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"611828f9e7ad9626e73c62e0c0fa5af440a60c715eb79f33db7c0965a1c729d5"} Sep 10 12:40:04.317373 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.317155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"bf625417e406e4535ce23e91c260d6867fd0437f467d4aa3e34f1562567fd3e2"} Sep 10 12:40:04.804261 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.804238 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:40:04.967796 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.967724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-registry-tls\") pod \"39f344d8-6329-495c-ab26-6e2b58e3f401\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " Sep 10 12:40:04.967796 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.967767 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fw82\" (UniqueName: \"kubernetes.io/projected/39f344d8-6329-495c-ab26-6e2b58e3f401-kube-api-access-2fw82\") pod \"39f344d8-6329-495c-ab26-6e2b58e3f401\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " Sep 10 12:40:04.967996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.967853 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39f344d8-6329-495c-ab26-6e2b58e3f401-feast-data\") pod \"39f344d8-6329-495c-ab26-6e2b58e3f401\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " Sep 10 12:40:04.967996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.967886 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-ui-tls\") pod \"39f344d8-6329-495c-ab26-6e2b58e3f401\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " Sep 10 12:40:04.967996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.967912 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-offline-tls\") pod \"39f344d8-6329-495c-ab26-6e2b58e3f401\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " Sep 10 12:40:04.967996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.967947 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls\") pod \"39f344d8-6329-495c-ab26-6e2b58e3f401\" (UID: \"39f344d8-6329-495c-ab26-6e2b58e3f401\") " Sep 10 12:40:04.968664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.968577 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f344d8-6329-495c-ab26-6e2b58e3f401-feast-data" (OuterVolumeSpecName: "feast-data") pod "39f344d8-6329-495c-ab26-6e2b58e3f401" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:40:04.970311 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.970274 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "39f344d8-6329-495c-ab26-6e2b58e3f401" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:40:04.970311 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.970296 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f344d8-6329-495c-ab26-6e2b58e3f401-kube-api-access-2fw82" (OuterVolumeSpecName: "kube-api-access-2fw82") pod "39f344d8-6329-495c-ab26-6e2b58e3f401" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401"). InnerVolumeSpecName "kube-api-access-2fw82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:40:04.970452 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.970327 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls" (OuterVolumeSpecName: "online-tls") pod "39f344d8-6329-495c-ab26-6e2b58e3f401" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:40:04.970572 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.970555 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "39f344d8-6329-495c-ab26-6e2b58e3f401" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:40:04.970662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:04.970612 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "39f344d8-6329-495c-ab26-6e2b58e3f401" (UID: "39f344d8-6329-495c-ab26-6e2b58e3f401"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:40:05.068525 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.068500 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/39f344d8-6329-495c-ab26-6e2b58e3f401-feast-data\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:40:05.068525 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.068522 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-ui-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:40:05.068525 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.068530 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-offline-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:40:05.068715 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.068538 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-online-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:40:05.068715 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.068546 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/39f344d8-6329-495c-ab26-6e2b58e3f401-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:40:05.068715 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.068553 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fw82\" (UniqueName: \"kubernetes.io/projected/39f344d8-6329-495c-ab26-6e2b58e3f401-kube-api-access-2fw82\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:40:05.324095 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.324067 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" Sep 10 12:40:05.324095 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.324075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg" event={"ID":"39f344d8-6329-495c-ab26-6e2b58e3f401","Type":"ContainerDied","Data":"cca02e29fb7078333c2a338040b463c4c930fecc10d5d828e3352d086587789d"} Sep 10 12:40:05.324644 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.324120 2562 scope.go:117] "RemoveContainer" containerID="92ad1f45d8d0e037bd2bca36bebbf033b790cbe5f25191a9a051990218ff8372" Sep 10 12:40:05.334301 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.334283 2562 scope.go:117] "RemoveContainer" containerID="611828f9e7ad9626e73c62e0c0fa5af440a60c715eb79f33db7c0965a1c729d5" Sep 10 12:40:05.342793 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.342772 2562 scope.go:117] "RemoveContainer" containerID="9c366d1d0a90a18b66c8043dfb6939d09526159277d96f072288999769856df9" Sep 10 12:40:05.351461 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.351441 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg"] Sep 10 12:40:05.351890 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.351876 2562 scope.go:117] "RemoveContainer" containerID="bf625417e406e4535ce23e91c260d6867fd0437f467d4aa3e34f1562567fd3e2" Sep 10 12:40:05.355742 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.355721 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-l2ggg"] Sep 10 12:40:05.359964 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.359947 2562 scope.go:117] "RemoveContainer" containerID="33e1ed7496d2480e342f7dfb2650a4f967b321086ebf01a85a1bfd1b432f10da" Sep 10 12:40:05.362702 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:40:05.362677 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39f344d8_6329_495c_ab26_6e2b58e3f401.slice/crio-cca02e29fb7078333c2a338040b463c4c930fecc10d5d828e3352d086587789d\": RecentStats: unable to find data in memory cache]" Sep 10 12:40:05.368623 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.368600 2562 scope.go:117] "RemoveContainer" containerID="6609ffeb7cd38077217081648c9a182afd4fe39ef09fdaa01364993248344de6" Sep 10 12:40:05.837866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:05.837829 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" path="/var/lib/kubelet/pods/39f344d8-6329-495c-ab26-6e2b58e3f401/volumes" Sep 10 12:40:59.719562 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719478 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9"] Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719855 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="offline" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719867 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="offline" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719881 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="registry" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719886 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="registry" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719894 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719900 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719915 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="feast-init" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719920 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="feast-init" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719926 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="feast-apply" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719930 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="feast-apply" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719936 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719941 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719988 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="online" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.719994 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="offline" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.720001 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="ui" Sep 10 12:40:59.720021 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.720008 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f344d8-6329-495c-ab26-6e2b58e3f401" containerName="registry" Sep 10 12:40:59.723143 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.723125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.727337 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.727319 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 10 12:40:59.727627 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.727571 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 10 12:40:59.727728 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.727633 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 10 12:40:59.727728 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.727643 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 10 12:40:59.727728 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.727666 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 10 12:40:59.727891 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.727848 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 10 12:40:59.728320 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.728301 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-x949m\"" Sep 10 12:40:59.728452 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.728377 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 10 12:40:59.756842 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.756822 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9"] Sep 10 12:40:59.776673 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.776649 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-offline-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.776758 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.776685 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-online-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.776758 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.776719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59v9m\" (UniqueName: \"kubernetes.io/projected/c333534b-0f45-427e-b415-0e0cff9a9666-kube-api-access-59v9m\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.776831 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.776812 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-ui-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.776865 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.776836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c333534b-0f45-427e-b415-0e0cff9a9666-feast-data\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.776899 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.776864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c333534b-0f45-427e-b415-0e0cff9a9666-registry-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877287 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-ui-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877412 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c333534b-0f45-427e-b415-0e0cff9a9666-feast-data\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877412 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877326 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c333534b-0f45-427e-b415-0e0cff9a9666-registry-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877412 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-offline-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877412 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-online-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877635 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59v9m\" (UniqueName: \"kubernetes.io/projected/c333534b-0f45-427e-b415-0e0cff9a9666-kube-api-access-59v9m\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.877685 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.877666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c333534b-0f45-427e-b415-0e0cff9a9666-feast-data\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.878174 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.878084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c333534b-0f45-427e-b415-0e0cff9a9666-registry-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.879971 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.879951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-ui-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.880056 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.879995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-offline-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.880056 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.880029 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-online-tls\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:40:59.890792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:40:59.890774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59v9m\" (UniqueName: \"kubernetes.io/projected/c333534b-0f45-427e-b415-0e0cff9a9666-kube-api-access-59v9m\") pod \"feast-simple-feast-remote-setup-dfdcf7995-7kdl9\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:00.032963 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:00.032942 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:00.330008 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:00.329983 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9"] Sep 10 12:41:00.331628 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:41:00.331574 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc333534b_0f45_427e_b415_0e0cff9a9666.slice/crio-f9c1bd9d24e0e86c240100886d6f13aceb7a759911fc172973bcfc0061760023 WatchSource:0}: Error finding container f9c1bd9d24e0e86c240100886d6f13aceb7a759911fc172973bcfc0061760023: Status 404 returned error can't find the container with id f9c1bd9d24e0e86c240100886d6f13aceb7a759911fc172973bcfc0061760023 Sep 10 12:41:00.548451 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:00.548417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerStarted","Data":"3f87e3fcacc1c328c0fa0fd72d79d2a6b097ccb35e017c723bdf113378de6cf2"} Sep 10 12:41:00.548623 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:00.548460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerStarted","Data":"f9c1bd9d24e0e86c240100886d6f13aceb7a759911fc172973bcfc0061760023"} Sep 10 12:41:05.571708 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:05.571676 2562 generic.go:356] "Generic (PLEG): container finished" podID="c333534b-0f45-427e-b415-0e0cff9a9666" containerID="3f87e3fcacc1c328c0fa0fd72d79d2a6b097ccb35e017c723bdf113378de6cf2" exitCode=0 Sep 10 12:41:05.572067 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:05.571744 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerDied","Data":"3f87e3fcacc1c328c0fa0fd72d79d2a6b097ccb35e017c723bdf113378de6cf2"} Sep 10 12:41:06.578206 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:06.578170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerStarted","Data":"5397c6ea47242bd3ec67846d807aabf00d71ce2223084fe9de8ef71a79a3cb6c"} Sep 10 12:41:30.681442 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:30.681402 2562 generic.go:356] "Generic (PLEG): container finished" podID="c333534b-0f45-427e-b415-0e0cff9a9666" containerID="5397c6ea47242bd3ec67846d807aabf00d71ce2223084fe9de8ef71a79a3cb6c" exitCode=0 Sep 10 12:41:30.682033 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:30.681481 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerDied","Data":"5397c6ea47242bd3ec67846d807aabf00d71ce2223084fe9de8ef71a79a3cb6c"} Sep 10 12:41:31.692566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:31.692533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerStarted","Data":"660eff2bd0f1d6af483ed1ebaca3deb5bb8c2d3033636d845d838fe5e2a233c8"} Sep 10 12:41:31.693048 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:31.692608 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerStarted","Data":"709da06eb0b9e9ee4e3de98814b13159e640eecf8e30ffa5534c621e68ff0e5b"} Sep 10 12:41:31.693048 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:31.692624 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerStarted","Data":"70a99453992d5c0811cbd917c5d089a8a247209ce39fb5c0586b6b43e6788247"} Sep 10 12:41:31.723143 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:31.723086 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podStartSLOduration=32.723065585 podStartE2EDuration="32.723065585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:41:31.721634856 +0000 UTC m=+658.347832433" watchObservedRunningTime="2026-09-10 12:41:31.723065585 +0000 UTC m=+658.349263161" Sep 10 12:41:33.033340 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:33.033302 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:33.033706 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:33.033362 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:33.033706 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:33.033382 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:33.035378 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:33.035341 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 10 12:41:33.035490 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:33.035368 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 10 12:41:33.035490 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:33.035415 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 10 12:41:36.033925 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:36.033872 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 10 12:41:36.034372 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:36.033866 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 10 12:41:36.034372 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:36.033866 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 10 12:41:39.034667 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.034570 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.035076 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.034771 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.035076 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.035028 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.035076 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.035069 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.035410 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.035383 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.035616 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.035600 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.056567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.056546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.733499 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.733468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:41:39.754527 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:41:39.754499 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:42:41.869960 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.869872 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9"] Sep 10 12:42:41.870443 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.870128 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" containerID="cri-o://70a99453992d5c0811cbd917c5d089a8a247209ce39fb5c0586b6b43e6788247" gracePeriod=30 Sep 10 12:42:41.870443 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.870201 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" containerID="cri-o://660eff2bd0f1d6af483ed1ebaca3deb5bb8c2d3033636d845d838fe5e2a233c8" gracePeriod=30 Sep 10 12:42:41.870443 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.870279 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="offline" containerID="cri-o://709da06eb0b9e9ee4e3de98814b13159e640eecf8e30ffa5534c621e68ff0e5b" gracePeriod=30 Sep 10 12:42:41.990888 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.990848 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4"] Sep 10 12:42:41.991122 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.991056 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="registry" containerID="cri-o://dc6636011719d5d6128df993d03de68f0bdf1454b8f2c1bf31e456cd9075ecc9" gracePeriod=30 Sep 10 12:42:41.991285 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.991126 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="offline" containerID="cri-o://9e10f54b5bfb278de6bf8d582fd9e399f6b18ff157f488e0576f9d5511a64233" gracePeriod=30 Sep 10 12:42:41.991285 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.991141 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" containerID="cri-o://6d746bafde52e7625e9bc03780febbef9ede55b9eec994bf481bdcfb96fcf936" gracePeriod=30 Sep 10 12:42:41.991285 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:41.991142 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" containerID="cri-o://07b1793818cbaa95aeae2ae6ec4b8c41c24dc70e93e0e6b4c4cd4c092bb19bc9" gracePeriod=30 Sep 10 12:42:42.998328 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:42.998249 2562 generic.go:356] "Generic (PLEG): container finished" podID="c333534b-0f45-427e-b415-0e0cff9a9666" containerID="660eff2bd0f1d6af483ed1ebaca3deb5bb8c2d3033636d845d838fe5e2a233c8" exitCode=0 Sep 10 12:42:42.998328 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:42.998277 2562 generic.go:356] "Generic (PLEG): container finished" podID="c333534b-0f45-427e-b415-0e0cff9a9666" containerID="70a99453992d5c0811cbd917c5d089a8a247209ce39fb5c0586b6b43e6788247" exitCode=0 Sep 10 12:42:42.998328 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:42.998310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerDied","Data":"660eff2bd0f1d6af483ed1ebaca3deb5bb8c2d3033636d845d838fe5e2a233c8"} Sep 10 12:42:42.998871 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:42.998359 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerDied","Data":"70a99453992d5c0811cbd917c5d089a8a247209ce39fb5c0586b6b43e6788247"} Sep 10 12:42:43.001501 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:43.001481 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerID="07b1793818cbaa95aeae2ae6ec4b8c41c24dc70e93e0e6b4c4cd4c092bb19bc9" exitCode=0 Sep 10 12:42:43.001501 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:43.001500 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerID="6d746bafde52e7625e9bc03780febbef9ede55b9eec994bf481bdcfb96fcf936" exitCode=0 Sep 10 12:42:43.001650 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:43.001553 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"07b1793818cbaa95aeae2ae6ec4b8c41c24dc70e93e0e6b4c4cd4c092bb19bc9"} Sep 10 12:42:43.001650 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:43.001607 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"6d746bafde52e7625e9bc03780febbef9ede55b9eec994bf481bdcfb96fcf936"} Sep 10 12:42:45.947364 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:45.947321 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 10 12:42:46.281834 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:46.281796 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 10 12:42:49.035330 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:49.035288 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 10 12:42:49.734028 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:49.733985 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 10 12:42:55.946813 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:55.946772 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 10 12:42:56.280929 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:56.280894 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 10 12:42:59.035547 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:59.035499 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 10 12:42:59.734627 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:42:59.734553 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 10 12:43:05.946915 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:05.946865 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.24:8443: connect: connection refused" Sep 10 12:43:05.947368 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:05.946998 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:43:06.280824 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:06.280786 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" probeResult="failure" output="Get \"https://10.134.0.24:6567/health\": dial tcp 10.134.0.24:6567: connect: connection refused" Sep 10 12:43:06.281001 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:06.280914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:43:09.035641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:09.035562 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.25:8443: connect: connection refused" Sep 10 12:43:09.036111 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:09.035721 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:43:09.734562 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:09.734517 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 10 12:43:09.734762 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:09.734689 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:43:12.121841 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.121811 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerID="9e10f54b5bfb278de6bf8d582fd9e399f6b18ff157f488e0576f9d5511a64233" exitCode=137 Sep 10 12:43:12.121841 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.121833 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerID="dc6636011719d5d6128df993d03de68f0bdf1454b8f2c1bf31e456cd9075ecc9" exitCode=137 Sep 10 12:43:12.122220 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.121879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"9e10f54b5bfb278de6bf8d582fd9e399f6b18ff157f488e0576f9d5511a64233"} Sep 10 12:43:12.122220 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.121929 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"dc6636011719d5d6128df993d03de68f0bdf1454b8f2c1bf31e456cd9075ecc9"} Sep 10 12:43:12.130240 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.130185 2562 generic.go:356] "Generic (PLEG): container finished" podID="c333534b-0f45-427e-b415-0e0cff9a9666" containerID="709da06eb0b9e9ee4e3de98814b13159e640eecf8e30ffa5534c621e68ff0e5b" exitCode=137 Sep 10 12:43:12.130240 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.130230 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerDied","Data":"709da06eb0b9e9ee4e3de98814b13159e640eecf8e30ffa5534c621e68ff0e5b"} Sep 10 12:43:12.516514 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.516487 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:43:12.586033 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.586010 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-offline-tls\") pod \"c333534b-0f45-427e-b415-0e0cff9a9666\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " Sep 10 12:43:12.586141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.586055 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59v9m\" (UniqueName: \"kubernetes.io/projected/c333534b-0f45-427e-b415-0e0cff9a9666-kube-api-access-59v9m\") pod \"c333534b-0f45-427e-b415-0e0cff9a9666\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " Sep 10 12:43:12.586141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.586097 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-ui-tls\") pod \"c333534b-0f45-427e-b415-0e0cff9a9666\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " Sep 10 12:43:12.586141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.586122 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-online-tls\") pod \"c333534b-0f45-427e-b415-0e0cff9a9666\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " Sep 10 12:43:12.586346 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.586317 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c333534b-0f45-427e-b415-0e0cff9a9666-feast-data\") pod \"c333534b-0f45-427e-b415-0e0cff9a9666\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " Sep 10 12:43:12.586492 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.586391 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c333534b-0f45-427e-b415-0e0cff9a9666-registry-tls\") pod \"c333534b-0f45-427e-b415-0e0cff9a9666\" (UID: \"c333534b-0f45-427e-b415-0e0cff9a9666\") " Sep 10 12:43:12.587240 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.587188 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c333534b-0f45-427e-b415-0e0cff9a9666-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c333534b-0f45-427e-b415-0e0cff9a9666" (UID: "c333534b-0f45-427e-b415-0e0cff9a9666"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 12:43:12.587344 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.587238 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c333534b-0f45-427e-b415-0e0cff9a9666-feast-data" (OuterVolumeSpecName: "feast-data") pod "c333534b-0f45-427e-b415-0e0cff9a9666" (UID: "c333534b-0f45-427e-b415-0e0cff9a9666"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:43:12.588594 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.588557 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c333534b-0f45-427e-b415-0e0cff9a9666" (UID: "c333534b-0f45-427e-b415-0e0cff9a9666"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.588679 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.588572 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c333534b-0f45-427e-b415-0e0cff9a9666-kube-api-access-59v9m" (OuterVolumeSpecName: "kube-api-access-59v9m") pod "c333534b-0f45-427e-b415-0e0cff9a9666" (UID: "c333534b-0f45-427e-b415-0e0cff9a9666"). InnerVolumeSpecName "kube-api-access-59v9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:43:12.588781 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.588759 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-online-tls" (OuterVolumeSpecName: "online-tls") pod "c333534b-0f45-427e-b415-0e0cff9a9666" (UID: "c333534b-0f45-427e-b415-0e0cff9a9666"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.588781 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.588770 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c333534b-0f45-427e-b415-0e0cff9a9666" (UID: "c333534b-0f45-427e-b415-0e0cff9a9666"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.636988 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.636963 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:43:12.686885 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.686833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-online-tls\") pod \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " Sep 10 12:43:12.686885 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.686857 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-offline-tls\") pod \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " Sep 10 12:43:12.686885 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.686877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cshl6\" (UniqueName: \"kubernetes.io/projected/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-kube-api-access-cshl6\") pod \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " Sep 10 12:43:12.687102 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.686895 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-feast-data\") pod \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " Sep 10 12:43:12.687102 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.686972 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-ui-tls\") pod \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " Sep 10 12:43:12.687102 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687020 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-registry-tls\") pod \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\" (UID: \"8bf6ed68-fd15-4b9c-b319-3e45709e8be5\") " Sep 10 12:43:12.687272 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687240 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-offline-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.687272 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687259 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59v9m\" (UniqueName: \"kubernetes.io/projected/c333534b-0f45-427e-b415-0e0cff9a9666-kube-api-access-59v9m\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.687375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687272 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-ui-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.687375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687285 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c333534b-0f45-427e-b415-0e0cff9a9666-online-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.687375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687298 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c333534b-0f45-427e-b415-0e0cff9a9666-feast-data\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.687375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687310 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/c333534b-0f45-427e-b415-0e0cff9a9666-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.687567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.687505 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-feast-data" (OuterVolumeSpecName: "feast-data") pod "8bf6ed68-fd15-4b9c-b319-3e45709e8be5" (UID: "8bf6ed68-fd15-4b9c-b319-3e45709e8be5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:43:12.689069 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.689048 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-kube-api-access-cshl6" (OuterVolumeSpecName: "kube-api-access-cshl6") pod "8bf6ed68-fd15-4b9c-b319-3e45709e8be5" (UID: "8bf6ed68-fd15-4b9c-b319-3e45709e8be5"). InnerVolumeSpecName "kube-api-access-cshl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:43:12.689533 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.689509 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "8bf6ed68-fd15-4b9c-b319-3e45709e8be5" (UID: "8bf6ed68-fd15-4b9c-b319-3e45709e8be5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.689663 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.689538 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "8bf6ed68-fd15-4b9c-b319-3e45709e8be5" (UID: "8bf6ed68-fd15-4b9c-b319-3e45709e8be5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.689733 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.689672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-online-tls" (OuterVolumeSpecName: "online-tls") pod "8bf6ed68-fd15-4b9c-b319-3e45709e8be5" (UID: "8bf6ed68-fd15-4b9c-b319-3e45709e8be5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.689733 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.689715 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8bf6ed68-fd15-4b9c-b319-3e45709e8be5" (UID: "8bf6ed68-fd15-4b9c-b319-3e45709e8be5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:43:12.787673 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.787640 2562 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-ui-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.787673 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.787667 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.787673 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.787678 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-online-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.787895 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.787688 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-offline-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.787895 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.787696 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshl6\" (UniqueName: \"kubernetes.io/projected/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-kube-api-access-cshl6\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:12.787895 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:12.787705 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf6ed68-fd15-4b9c-b319-3e45709e8be5-feast-data\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:43:13.137204 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.137179 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" Sep 10 12:43:13.137688 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.137198 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4" event={"ID":"8bf6ed68-fd15-4b9c-b319-3e45709e8be5","Type":"ContainerDied","Data":"0edf0e259eb22edc4d9ac99b59c19648b9b0785be167cbb7e666fb27270f6561"} Sep 10 12:43:13.137688 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.137263 2562 scope.go:117] "RemoveContainer" containerID="07b1793818cbaa95aeae2ae6ec4b8c41c24dc70e93e0e6b4c4cd4c092bb19bc9" Sep 10 12:43:13.140114 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.140086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" event={"ID":"c333534b-0f45-427e-b415-0e0cff9a9666","Type":"ContainerDied","Data":"f9c1bd9d24e0e86c240100886d6f13aceb7a759911fc172973bcfc0061760023"} Sep 10 12:43:13.140229 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.140130 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9" Sep 10 12:43:13.147691 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.147671 2562 scope.go:117] "RemoveContainer" containerID="9e10f54b5bfb278de6bf8d582fd9e399f6b18ff157f488e0576f9d5511a64233" Sep 10 12:43:13.155371 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.155355 2562 scope.go:117] "RemoveContainer" containerID="6d746bafde52e7625e9bc03780febbef9ede55b9eec994bf481bdcfb96fcf936" Sep 10 12:43:13.162943 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.162897 2562 scope.go:117] "RemoveContainer" containerID="dc6636011719d5d6128df993d03de68f0bdf1454b8f2c1bf31e456cd9075ecc9" Sep 10 12:43:13.164306 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.164287 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4"] Sep 10 12:43:13.170457 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.170437 2562 scope.go:117] "RemoveContainer" containerID="58c87cd9e5f692c473c85737742363dfee25fdc88b5930ac8975884f962f89de" Sep 10 12:43:13.171483 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.171468 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-789ddffb9f-w7lq4"] Sep 10 12:43:13.177875 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.177851 2562 scope.go:117] "RemoveContainer" containerID="8773053f582ebd8a4a2c0fd016a0f79dfc67cb685c36f17e7e46288746b1c242" Sep 10 12:43:13.184320 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.184302 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9"] Sep 10 12:43:13.185223 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.185202 2562 scope.go:117] "RemoveContainer" containerID="660eff2bd0f1d6af483ed1ebaca3deb5bb8c2d3033636d845d838fe5e2a233c8" Sep 10 12:43:13.188264 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.188245 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-dfdcf7995-7kdl9"] Sep 10 12:43:13.192476 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.192460 2562 scope.go:117] "RemoveContainer" containerID="709da06eb0b9e9ee4e3de98814b13159e640eecf8e30ffa5534c621e68ff0e5b" Sep 10 12:43:13.199309 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.199293 2562 scope.go:117] "RemoveContainer" containerID="70a99453992d5c0811cbd917c5d089a8a247209ce39fb5c0586b6b43e6788247" Sep 10 12:43:13.206549 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.206532 2562 scope.go:117] "RemoveContainer" containerID="5397c6ea47242bd3ec67846d807aabf00d71ce2223084fe9de8ef71a79a3cb6c" Sep 10 12:43:13.213300 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.213282 2562 scope.go:117] "RemoveContainer" containerID="3f87e3fcacc1c328c0fa0fd72d79d2a6b097ccb35e017c723bdf113378de6cf2" Sep 10 12:43:13.839338 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.839303 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" path="/var/lib/kubelet/pods/8bf6ed68-fd15-4b9c-b319-3e45709e8be5/volumes" Sep 10 12:43:13.839945 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:13.839928 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" path="/var/lib/kubelet/pods/c333534b-0f45-427e-b415-0e0cff9a9666/volumes" Sep 10 12:43:25.101002 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.100970 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-msmlk"] Sep 10 12:43:25.101410 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.101289 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" Sep 10 12:43:25.101410 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.101300 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" Sep 10 12:43:25.101597 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.101561 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" Sep 10 12:43:25.108319 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108292 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" Sep 10 12:43:25.108533 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108518 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="feast-apply" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108641 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="feast-apply" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108678 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="offline" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108688 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="offline" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108715 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="feast-apply" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108724 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="feast-apply" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108735 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108751 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108762 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="offline" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108771 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="offline" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108792 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="registry" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108802 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="registry" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108815 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" Sep 10 12:43:25.108662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108824 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108868 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="feast-init" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108891 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="feast-init" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108914 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="feast-init" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.108923 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="feast-init" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109351 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="registry" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109371 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="offline" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109382 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="online" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109392 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="online" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109401 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c333534b-0f45-427e-b415-0e0cff9a9666" containerName="ui" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109410 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="offline" Sep 10 12:43:25.109438 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.109420 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf6ed68-fd15-4b9c-b319-3e45709e8be5" containerName="ui" Sep 10 12:43:25.115035 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.115005 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.115984 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.115965 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-msmlk"] Sep 10 12:43:25.117641 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.117618 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 10 12:43:25.118247 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.118229 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 10 12:43:25.118624 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.118611 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-6dlrg\"" Sep 10 12:43:25.118688 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.118627 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 10 12:43:25.176236 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.176212 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffdk5\" (UniqueName: \"kubernetes.io/projected/e2872db1-2ae1-4519-b501-3a368b32f5b6-kube-api-access-ffdk5\") pod \"postgres-864846d4f5-msmlk\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.176349 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.176263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e2872db1-2ae1-4519-b501-3a368b32f5b6-postgresdata\") pod \"postgres-864846d4f5-msmlk\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.276664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.276640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffdk5\" (UniqueName: \"kubernetes.io/projected/e2872db1-2ae1-4519-b501-3a368b32f5b6-kube-api-access-ffdk5\") pod \"postgres-864846d4f5-msmlk\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.276784 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.276698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e2872db1-2ae1-4519-b501-3a368b32f5b6-postgresdata\") pod \"postgres-864846d4f5-msmlk\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.277062 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.277042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e2872db1-2ae1-4519-b501-3a368b32f5b6-postgresdata\") pod \"postgres-864846d4f5-msmlk\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.287149 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.287125 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffdk5\" (UniqueName: \"kubernetes.io/projected/e2872db1-2ae1-4519-b501-3a368b32f5b6-kube-api-access-ffdk5\") pod \"postgres-864846d4f5-msmlk\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.431687 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.431626 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:43:25.700502 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.700430 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-msmlk"] Sep 10 12:43:25.703806 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:43:25.703780 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2872db1_2ae1_4519_b501_3a368b32f5b6.slice/crio-dddabb7deed151cba6ebf03f83ead53c72ca2638703997d0ac36521b02eee707 WatchSource:0}: Error finding container dddabb7deed151cba6ebf03f83ead53c72ca2638703997d0ac36521b02eee707: Status 404 returned error can't find the container with id dddabb7deed151cba6ebf03f83ead53c72ca2638703997d0ac36521b02eee707 Sep 10 12:43:25.705758 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:25.705739 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 12:43:26.195740 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:26.195708 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-msmlk" event={"ID":"e2872db1-2ae1-4519-b501-3a368b32f5b6","Type":"ContainerStarted","Data":"dddabb7deed151cba6ebf03f83ead53c72ca2638703997d0ac36521b02eee707"} Sep 10 12:43:32.225020 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:32.224984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-msmlk" event={"ID":"e2872db1-2ae1-4519-b501-3a368b32f5b6","Type":"ContainerStarted","Data":"49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867"} Sep 10 12:43:35.996605 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:35.996526 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-msmlk" podStartSLOduration=5.47842019 podStartE2EDuration="10.996505892s" totalImagesPullingTime="5.518085702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:32.2531951 +0000 UTC m=+778.879392798" watchObservedRunningTime="2026-09-10 12:43:35.996505892 +0000 UTC m=+782.622703469" Sep 10 12:43:35.997644 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:35.997627 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc"] Sep 10 12:43:36.004235 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.004214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.010863 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.008663 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 10 12:43:36.010863 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.008722 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 10 12:43:36.011121 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.011062 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 10 12:43:36.011566 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.011320 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-p864p\"" Sep 10 12:43:36.011680 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.011625 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 10 12:43:36.024474 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.024454 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc"] Sep 10 12:43:36.064277 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.064254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-online-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.064375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.064290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhf6v\" (UniqueName: \"kubernetes.io/projected/e20148df-84c7-45e4-ab5c-5985e4d74856-kube-api-access-lhf6v\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.064375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.064320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-offline-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.064375 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.064357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-registry-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.064478 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.064395 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e20148df-84c7-45e4-ab5c-5985e4d74856-feast-data\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.165306 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.165281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-registry-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.165427 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.165318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e20148df-84c7-45e4-ab5c-5985e4d74856-feast-data\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.165427 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.165363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-online-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.165427 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.165402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhf6v\" (UniqueName: \"kubernetes.io/projected/e20148df-84c7-45e4-ab5c-5985e4d74856-kube-api-access-lhf6v\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.165612 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.165444 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-offline-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.165833 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.165814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e20148df-84c7-45e4-ab5c-5985e4d74856-feast-data\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.167905 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.167886 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-registry-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.168416 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.168396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-online-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.168465 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.168407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-offline-tls\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.174075 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.174054 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhf6v\" (UniqueName: \"kubernetes.io/projected/e20148df-84c7-45e4-ab5c-5985e4d74856-kube-api-access-lhf6v\") pod \"feast-credit-scoring-5546c5f98b-vv6cc\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.314929 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.314898 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:36.600960 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:36.600935 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc"] Sep 10 12:43:36.602673 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:43:36.602648 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20148df_84c7_45e4_ab5c_5985e4d74856.slice/crio-ddda25aecdbea7d48f514b9c66b50c8de87ca37d6327a1da3fcfdd5ec38e3faf WatchSource:0}: Error finding container ddda25aecdbea7d48f514b9c66b50c8de87ca37d6327a1da3fcfdd5ec38e3faf: Status 404 returned error can't find the container with id ddda25aecdbea7d48f514b9c66b50c8de87ca37d6327a1da3fcfdd5ec38e3faf Sep 10 12:43:37.249977 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:37.249944 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerStarted","Data":"8b09530a692d1d5b8b2ee212c5956a4e35cb8e11562457fa48b709eb2e616ec9"} Sep 10 12:43:37.250423 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:37.250008 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerStarted","Data":"ddda25aecdbea7d48f514b9c66b50c8de87ca37d6327a1da3fcfdd5ec38e3faf"} Sep 10 12:43:38.255517 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:38.255478 2562 generic.go:356] "Generic (PLEG): container finished" podID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerID="8b09530a692d1d5b8b2ee212c5956a4e35cb8e11562457fa48b709eb2e616ec9" exitCode=0 Sep 10 12:43:38.255892 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:38.255566 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerDied","Data":"8b09530a692d1d5b8b2ee212c5956a4e35cb8e11562457fa48b709eb2e616ec9"} Sep 10 12:43:39.261941 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:39.261891 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerStarted","Data":"0c639ee8b1c77cc4a37e3b738ae7cebaf3360f5a0c13a696515c1e94650b7fa9"} Sep 10 12:43:45.288377 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:45.288341 2562 generic.go:356] "Generic (PLEG): container finished" podID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerID="0c639ee8b1c77cc4a37e3b738ae7cebaf3360f5a0c13a696515c1e94650b7fa9" exitCode=0 Sep 10 12:43:45.288377 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:45.288380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerDied","Data":"0c639ee8b1c77cc4a37e3b738ae7cebaf3360f5a0c13a696515c1e94650b7fa9"} Sep 10 12:43:46.297121 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.297087 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerStarted","Data":"a0ec1fc539cae346822367ad7755b1d855934da242b0b41f660470eb11af9ddf"} Sep 10 12:43:46.297618 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.297144 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerStarted","Data":"b9772837d7e86a1feda64814e24c3daada354e5070b4aa9171beee1002340afb"} Sep 10 12:43:46.297618 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.297162 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerStarted","Data":"128a69291e8c220f98f406e64cf598678aacbac83e34d18e3fa87ff9834fcdab"} Sep 10 12:43:46.315733 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.315706 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:46.315870 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.315759 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:46.315870 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.315777 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:46.326963 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:46.326911 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podStartSLOduration=11.326893275 podStartE2EDuration="11.326893275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:46.326478527 +0000 UTC m=+792.952676116" watchObservedRunningTime="2026-09-10 12:43:46.326893275 +0000 UTC m=+792.953090854" Sep 10 12:43:48.315435 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:48.315400 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:48.315912 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:48.315511 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:48.315912 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:48.315539 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:48.316823 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:48.316791 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 10 12:43:48.316952 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:48.316817 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 10 12:43:48.316952 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:48.316828 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 10 12:43:51.316073 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:51.316029 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 10 12:43:51.317005 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:51.316029 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 10 12:43:51.317196 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:51.316069 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.27:8816: connect: connection refused" Sep 10 12:43:54.315981 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:54.315930 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 10 12:43:54.316993 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:54.316969 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:54.317777 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:54.317740 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:54.322191 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:54.322169 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:54.346329 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:54.346307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:43:57.315707 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:43:57.315617 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.27:6571: connect: connection refused" Sep 10 12:44:00.316227 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:44:00.316200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:44:00.317003 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:44:00.316982 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:45:04.589727 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.589694 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc"] Sep 10 12:45:04.590245 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.589931 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" containerID="cri-o://128a69291e8c220f98f406e64cf598678aacbac83e34d18e3fa87ff9834fcdab" gracePeriod=30 Sep 10 12:45:04.590245 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.590000 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" containerID="cri-o://b9772837d7e86a1feda64814e24c3daada354e5070b4aa9171beee1002340afb" gracePeriod=30 Sep 10 12:45:04.590245 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.589983 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="offline" containerID="cri-o://a0ec1fc539cae346822367ad7755b1d855934da242b0b41f660470eb11af9ddf" gracePeriod=30 Sep 10 12:45:04.616939 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.616907 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-msmlk"] Sep 10 12:45:04.617102 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.617083 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-msmlk" podUID="e2872db1-2ae1-4519-b501-3a368b32f5b6" containerName="postgres" containerID="cri-o://49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867" gracePeriod=30 Sep 10 12:45:04.866452 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.866430 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:45:04.978908 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.978880 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e2872db1-2ae1-4519-b501-3a368b32f5b6-postgresdata\") pod \"e2872db1-2ae1-4519-b501-3a368b32f5b6\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " Sep 10 12:45:04.979091 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.978949 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffdk5\" (UniqueName: \"kubernetes.io/projected/e2872db1-2ae1-4519-b501-3a368b32f5b6-kube-api-access-ffdk5\") pod \"e2872db1-2ae1-4519-b501-3a368b32f5b6\" (UID: \"e2872db1-2ae1-4519-b501-3a368b32f5b6\") " Sep 10 12:45:04.981041 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:04.981011 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2872db1-2ae1-4519-b501-3a368b32f5b6-kube-api-access-ffdk5" (OuterVolumeSpecName: "kube-api-access-ffdk5") pod "e2872db1-2ae1-4519-b501-3a368b32f5b6" (UID: "e2872db1-2ae1-4519-b501-3a368b32f5b6"). InnerVolumeSpecName "kube-api-access-ffdk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:45:05.011407 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.011378 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2872db1-2ae1-4519-b501-3a368b32f5b6-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "e2872db1-2ae1-4519-b501-3a368b32f5b6" (UID: "e2872db1-2ae1-4519-b501-3a368b32f5b6"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:45:05.079846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.079808 2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e2872db1-2ae1-4519-b501-3a368b32f5b6-postgresdata\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:05.079846 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.079840 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffdk5\" (UniqueName: \"kubernetes.io/projected/e2872db1-2ae1-4519-b501-3a368b32f5b6-kube-api-access-ffdk5\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:05.646864 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.646782 2562 generic.go:356] "Generic (PLEG): container finished" podID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerID="b9772837d7e86a1feda64814e24c3daada354e5070b4aa9171beee1002340afb" exitCode=0 Sep 10 12:45:05.646864 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.646855 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerDied","Data":"b9772837d7e86a1feda64814e24c3daada354e5070b4aa9171beee1002340afb"} Sep 10 12:45:05.648130 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.648108 2562 generic.go:356] "Generic (PLEG): container finished" podID="e2872db1-2ae1-4519-b501-3a368b32f5b6" containerID="49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867" exitCode=0 Sep 10 12:45:05.648205 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.648155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-msmlk" event={"ID":"e2872db1-2ae1-4519-b501-3a368b32f5b6","Type":"ContainerDied","Data":"49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867"} Sep 10 12:45:05.648205 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.648189 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-msmlk" event={"ID":"e2872db1-2ae1-4519-b501-3a368b32f5b6","Type":"ContainerDied","Data":"dddabb7deed151cba6ebf03f83ead53c72ca2638703997d0ac36521b02eee707"} Sep 10 12:45:05.648280 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.648205 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-msmlk" Sep 10 12:45:05.648319 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.648207 2562 scope.go:117] "RemoveContainer" containerID="49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867" Sep 10 12:45:05.659177 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.659163 2562 scope.go:117] "RemoveContainer" containerID="49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867" Sep 10 12:45:05.659440 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:45:05.659420 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867\": container with ID starting with 49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867 not found: ID does not exist" containerID="49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867" Sep 10 12:45:05.659506 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.659446 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867"} err="failed to get container status \"49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867\": rpc error: code = NotFound desc = could not find container \"49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867\": container with ID starting with 49c63953421697d016517c0c6c302e9ee5e769e3c4aeb1e0d89001cc8eebf867 not found: ID does not exist" Sep 10 12:45:05.684060 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.684037 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-msmlk"] Sep 10 12:45:05.701038 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.701013 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-msmlk"] Sep 10 12:45:05.838175 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:05.838143 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2872db1-2ae1-4519-b501-3a368b32f5b6" path="/var/lib/kubelet/pods/e2872db1-2ae1-4519-b501-3a368b32f5b6/volumes" Sep 10 12:45:14.343721 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:14.343681 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 10 12:45:24.343266 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:24.343223 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 10 12:45:34.342746 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:34.342703 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" probeResult="failure" output="Get \"https://10.134.0.27:6567/health\": dial tcp 10.134.0.27:6567: connect: connection refused" Sep 10 12:45:34.343142 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:34.342846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:45:34.773989 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:34.773956 2562 generic.go:356] "Generic (PLEG): container finished" podID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerID="a0ec1fc539cae346822367ad7755b1d855934da242b0b41f660470eb11af9ddf" exitCode=137 Sep 10 12:45:34.773989 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:34.773982 2562 generic.go:356] "Generic (PLEG): container finished" podID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerID="128a69291e8c220f98f406e64cf598678aacbac83e34d18e3fa87ff9834fcdab" exitCode=137 Sep 10 12:45:34.774141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:34.774032 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerDied","Data":"a0ec1fc539cae346822367ad7755b1d855934da242b0b41f660470eb11af9ddf"} Sep 10 12:45:34.774141 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:34.774089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerDied","Data":"128a69291e8c220f98f406e64cf598678aacbac83e34d18e3fa87ff9834fcdab"} Sep 10 12:45:35.243636 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.243615 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:45:35.285996 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.285973 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-registry-tls\") pod \"e20148df-84c7-45e4-ab5c-5985e4d74856\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " Sep 10 12:45:35.286086 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.286002 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-online-tls\") pod \"e20148df-84c7-45e4-ab5c-5985e4d74856\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " Sep 10 12:45:35.286086 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.286038 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e20148df-84c7-45e4-ab5c-5985e4d74856-feast-data\") pod \"e20148df-84c7-45e4-ab5c-5985e4d74856\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " Sep 10 12:45:35.286086 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.286082 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-offline-tls\") pod \"e20148df-84c7-45e4-ab5c-5985e4d74856\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " Sep 10 12:45:35.286255 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.286129 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhf6v\" (UniqueName: \"kubernetes.io/projected/e20148df-84c7-45e4-ab5c-5985e4d74856-kube-api-access-lhf6v\") pod \"e20148df-84c7-45e4-ab5c-5985e4d74856\" (UID: \"e20148df-84c7-45e4-ab5c-5985e4d74856\") " Sep 10 12:45:35.288515 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.288484 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e20148df-84c7-45e4-ab5c-5985e4d74856" (UID: "e20148df-84c7-45e4-ab5c-5985e4d74856"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:45:35.289577 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.289550 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20148df-84c7-45e4-ab5c-5985e4d74856-kube-api-access-lhf6v" (OuterVolumeSpecName: "kube-api-access-lhf6v") pod "e20148df-84c7-45e4-ab5c-5985e4d74856" (UID: "e20148df-84c7-45e4-ab5c-5985e4d74856"). InnerVolumeSpecName "kube-api-access-lhf6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:45:35.289577 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.289553 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "e20148df-84c7-45e4-ab5c-5985e4d74856" (UID: "e20148df-84c7-45e4-ab5c-5985e4d74856"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:45:35.289796 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.289770 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-online-tls" (OuterVolumeSpecName: "online-tls") pod "e20148df-84c7-45e4-ab5c-5985e4d74856" (UID: "e20148df-84c7-45e4-ab5c-5985e4d74856"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:45:35.295750 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.295725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20148df-84c7-45e4-ab5c-5985e4d74856-feast-data" (OuterVolumeSpecName: "feast-data") pod "e20148df-84c7-45e4-ab5c-5985e4d74856" (UID: "e20148df-84c7-45e4-ab5c-5985e4d74856"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:45:35.387218 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.387162 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:35.387218 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.387214 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-online-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:35.387528 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.387226 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/e20148df-84c7-45e4-ab5c-5985e4d74856-feast-data\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:35.387528 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.387234 2562 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/e20148df-84c7-45e4-ab5c-5985e4d74856-offline-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:35.387528 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.387242 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhf6v\" (UniqueName: \"kubernetes.io/projected/e20148df-84c7-45e4-ab5c-5985e4d74856-kube-api-access-lhf6v\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:45:35.781363 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.781328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" event={"ID":"e20148df-84c7-45e4-ab5c-5985e4d74856","Type":"ContainerDied","Data":"ddda25aecdbea7d48f514b9c66b50c8de87ca37d6327a1da3fcfdd5ec38e3faf"} Sep 10 12:45:35.781599 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.781393 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc" Sep 10 12:45:35.781599 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.781419 2562 scope.go:117] "RemoveContainer" containerID="a0ec1fc539cae346822367ad7755b1d855934da242b0b41f660470eb11af9ddf" Sep 10 12:45:35.795049 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.795024 2562 scope.go:117] "RemoveContainer" containerID="b9772837d7e86a1feda64814e24c3daada354e5070b4aa9171beee1002340afb" Sep 10 12:45:35.803480 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.803461 2562 scope.go:117] "RemoveContainer" containerID="128a69291e8c220f98f406e64cf598678aacbac83e34d18e3fa87ff9834fcdab" Sep 10 12:45:35.811323 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.811302 2562 scope.go:117] "RemoveContainer" containerID="0c639ee8b1c77cc4a37e3b738ae7cebaf3360f5a0c13a696515c1e94650b7fa9" Sep 10 12:45:35.819574 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.819543 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc"] Sep 10 12:45:35.819664 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.819617 2562 scope.go:117] "RemoveContainer" containerID="8b09530a692d1d5b8b2ee212c5956a4e35cb8e11562457fa48b709eb2e616ec9" Sep 10 12:45:35.826510 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.826482 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-5546c5f98b-vv6cc"] Sep 10 12:45:35.837908 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:45:35.837884 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" path="/var/lib/kubelet/pods/e20148df-84c7-45e4-ab5c-5985e4d74856/volumes" Sep 10 12:47:46.402614 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.402559 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-dv7lf"] Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403056 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="offline" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403076 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="offline" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403099 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403107 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403130 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2872db1-2ae1-4519-b501-3a368b32f5b6" containerName="postgres" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403138 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2872db1-2ae1-4519-b501-3a368b32f5b6" containerName="postgres" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403151 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="feast-init" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403160 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="feast-init" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403174 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="feast-apply" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403182 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="feast-apply" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403193 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" Sep 10 12:47:46.403228 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403202 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" Sep 10 12:47:46.403852 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403277 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="registry" Sep 10 12:47:46.403852 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403289 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="online" Sep 10 12:47:46.403852 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403301 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2872db1-2ae1-4519-b501-3a368b32f5b6" containerName="postgres" Sep 10 12:47:46.403852 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.403309 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20148df-84c7-45e4-ab5c-5985e4d74856" containerName="offline" Sep 10 12:47:46.406565 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.406544 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.410391 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.410363 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 10 12:47:46.410391 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.410387 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 10 12:47:46.410601 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.410372 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-sl2bt\"" Sep 10 12:47:46.411231 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.411208 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 10 12:47:46.419676 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.419653 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-dv7lf"] Sep 10 12:47:46.557381 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.557356 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b75529bb-7d83-43a8-9111-a418e792ee5f-postgresdata\") pod \"postgres-864846d4f5-dv7lf\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.557526 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.557384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99bnt\" (UniqueName: \"kubernetes.io/projected/b75529bb-7d83-43a8-9111-a418e792ee5f-kube-api-access-99bnt\") pod \"postgres-864846d4f5-dv7lf\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.657773 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.657700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b75529bb-7d83-43a8-9111-a418e792ee5f-postgresdata\") pod \"postgres-864846d4f5-dv7lf\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.657773 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.657741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99bnt\" (UniqueName: \"kubernetes.io/projected/b75529bb-7d83-43a8-9111-a418e792ee5f-kube-api-access-99bnt\") pod \"postgres-864846d4f5-dv7lf\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.658064 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.658039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b75529bb-7d83-43a8-9111-a418e792ee5f-postgresdata\") pod \"postgres-864846d4f5-dv7lf\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.667192 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.667172 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99bnt\" (UniqueName: \"kubernetes.io/projected/b75529bb-7d83-43a8-9111-a418e792ee5f-kube-api-access-99bnt\") pod \"postgres-864846d4f5-dv7lf\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:46.728882 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:46.728859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:47:47.002698 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:47.002625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-dv7lf"] Sep 10 12:47:47.005779 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:47:47.005756 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb75529bb_7d83_43a8_9111_a418e792ee5f.slice/crio-6553af0c6ad2ba7a75b079541f286dca5e111c14564cf5b8b08f91e0a0cb5bc6 WatchSource:0}: Error finding container 6553af0c6ad2ba7a75b079541f286dca5e111c14564cf5b8b08f91e0a0cb5bc6: Status 404 returned error can't find the container with id 6553af0c6ad2ba7a75b079541f286dca5e111c14564cf5b8b08f91e0a0cb5bc6 Sep 10 12:47:47.296775 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:47.296739 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" event={"ID":"b75529bb-7d83-43a8-9111-a418e792ee5f","Type":"ContainerStarted","Data":"1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f"} Sep 10 12:47:47.296953 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:47.296782 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" event={"ID":"b75529bb-7d83-43a8-9111-a418e792ee5f","Type":"ContainerStarted","Data":"6553af0c6ad2ba7a75b079541f286dca5e111c14564cf5b8b08f91e0a0cb5bc6"} Sep 10 12:47:47.313660 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:47.313607 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" podStartSLOduration=1.313569515 podStartE2EDuration="1.313569515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:47:47.313014287 +0000 UTC m=+1033.939211863" watchObservedRunningTime="2026-09-10 12:47:47.313569515 +0000 UTC m=+1033.939767090" Sep 10 12:47:52.134130 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.134099 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj"] Sep 10 12:47:52.137540 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.137526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.162250 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.162224 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 10 12:47:52.162345 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.162333 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 10 12:47:52.162488 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.162476 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 10 12:47:52.163869 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.163850 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj"] Sep 10 12:47:52.166526 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.166511 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-rxxxm\"" Sep 10 12:47:52.301475 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.301452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-online-tls\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.301634 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.301486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc55cae3-f3ca-4e44-81d9-32e13e172a56-feast-data\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.301634 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.301513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-registry-tls\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.301634 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.301622 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9sw\" (UniqueName: \"kubernetes.io/projected/bc55cae3-f3ca-4e44-81d9-32e13e172a56-kube-api-access-kf9sw\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.402676 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.402613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-registry-tls\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.402676 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.402663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9sw\" (UniqueName: \"kubernetes.io/projected/bc55cae3-f3ca-4e44-81d9-32e13e172a56-kube-api-access-kf9sw\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.402833 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.402700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-online-tls\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.402833 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.402727 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc55cae3-f3ca-4e44-81d9-32e13e172a56-feast-data\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.403111 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.403089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc55cae3-f3ca-4e44-81d9-32e13e172a56-feast-data\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.405388 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.405367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-online-tls\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.405490 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.405427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-registry-tls\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.410871 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.410849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9sw\" (UniqueName: \"kubernetes.io/projected/bc55cae3-f3ca-4e44-81d9-32e13e172a56-kube-api-access-kf9sw\") pod \"feast-credit-scoring-7ff84d6969-rhghj\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.445866 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.445843 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:47:52.731757 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:52.731683 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj"] Sep 10 12:47:52.736157 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:47:52.736129 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc55cae3_f3ca_4e44_81d9_32e13e172a56.slice/crio-16d0fcc29c2c3eedfdc2c60c749152033d2c36ca60e5f32dde2dbb60eb3fa6c5 WatchSource:0}: Error finding container 16d0fcc29c2c3eedfdc2c60c749152033d2c36ca60e5f32dde2dbb60eb3fa6c5: Status 404 returned error can't find the container with id 16d0fcc29c2c3eedfdc2c60c749152033d2c36ca60e5f32dde2dbb60eb3fa6c5 Sep 10 12:47:53.322098 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:53.322062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerStarted","Data":"0ac4ced518e9e5039637a727b39c8ff3c317094b30011557ac4f9cb48b730227"} Sep 10 12:47:53.322503 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:53.322121 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerStarted","Data":"16d0fcc29c2c3eedfdc2c60c749152033d2c36ca60e5f32dde2dbb60eb3fa6c5"} Sep 10 12:47:54.328194 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:54.328152 2562 generic.go:356] "Generic (PLEG): container finished" podID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerID="0ac4ced518e9e5039637a727b39c8ff3c317094b30011557ac4f9cb48b730227" exitCode=0 Sep 10 12:47:54.328660 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:54.328206 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerDied","Data":"0ac4ced518e9e5039637a727b39c8ff3c317094b30011557ac4f9cb48b730227"} Sep 10 12:47:55.334106 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:47:55.334066 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerStarted","Data":"9df417805f03e396f132b1d9b7a1bb0efd8e5880ebc347666dd5a2de53ec3dd1"} Sep 10 12:48:01.358939 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:01.358902 2562 generic.go:356] "Generic (PLEG): container finished" podID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerID="9df417805f03e396f132b1d9b7a1bb0efd8e5880ebc347666dd5a2de53ec3dd1" exitCode=0 Sep 10 12:48:01.359285 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:01.358971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerDied","Data":"9df417805f03e396f132b1d9b7a1bb0efd8e5880ebc347666dd5a2de53ec3dd1"} Sep 10 12:48:02.366302 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:02.366258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerStarted","Data":"a54f0adef70fc02840c43bbc60e587f896c7469dc516b926166a2881d0ecbde9"} Sep 10 12:48:02.366731 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:02.366330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerStarted","Data":"aec6b02b9c3884fbe402982683148c685ef39901a355664e115e03458fe50a06"} Sep 10 12:48:02.393814 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:02.393763 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podStartSLOduration=10.393744311 podStartE2EDuration="10.393744311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:02.392727445 +0000 UTC m=+1049.018925023" watchObservedRunningTime="2026-09-10 12:48:02.393744311 +0000 UTC m=+1049.019941887" Sep 10 12:48:02.446746 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:02.446720 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:02.446868 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:02.446818 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:04.446542 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:04.446513 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:04.446906 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:04.446566 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:04.448077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:04.448049 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 10 12:48:04.448186 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:04.448097 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 10 12:48:07.446858 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:07.446812 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 10 12:48:07.447333 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:07.446891 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 10 12:48:10.446845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:10.446795 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 10 12:48:10.452845 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:10.452822 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:10.456032 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:10.456011 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:13.447153 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:13.447127 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:13.448007 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:13.447991 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:48:17.938728 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.938697 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9"] Sep 10 12:48:17.943400 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.943383 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:17.945836 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.945818 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 10 12:48:17.946422 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.946403 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 10 12:48:17.946669 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.946654 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 10 12:48:17.946669 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.946667 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-fknx7\"" Sep 10 12:48:17.956485 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:17.956464 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9"] Sep 10 12:48:18.014818 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.014797 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-registry-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.014947 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.014843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.014947 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.014869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8833229f-9604-4dae-95a6-6cf6a6d617c4-feast-data\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.014947 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.014893 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xp9v\" (UniqueName: \"kubernetes.io/projected/8833229f-9604-4dae-95a6-6cf6a6d617c4-kube-api-access-4xp9v\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.115792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.115766 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.115902 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.115795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8833229f-9604-4dae-95a6-6cf6a6d617c4-feast-data\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.115902 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.115813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xp9v\" (UniqueName: \"kubernetes.io/projected/8833229f-9604-4dae-95a6-6cf6a6d617c4-kube-api-access-4xp9v\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.115902 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.115864 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-registry-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.116068 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:48:18.115927 2562 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 10 12:48:18.116068 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:48:18.116012 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls podName:8833229f-9604-4dae-95a6-6cf6a6d617c4 nodeName:}" failed. No retries permitted until 2026-09-10 12:48:18.615989459 +0000 UTC m=+1065.242187015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls") pod "feast-driver-ranking-75d7946bc4-jwxn9" (UID: "8833229f-9604-4dae-95a6-6cf6a6d617c4") : secret "feast-driver-ranking-online-tls" not found Sep 10 12:48:18.116212 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.116132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8833229f-9604-4dae-95a6-6cf6a6d617c4-feast-data\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.118238 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.118220 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-registry-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.128112 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.128085 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xp9v\" (UniqueName: \"kubernetes.io/projected/8833229f-9604-4dae-95a6-6cf6a6d617c4-kube-api-access-4xp9v\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.618803 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:18.618760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:18.618980 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:48:18.618921 2562 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 10 12:48:18.619031 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:48:18.619009 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls podName:8833229f-9604-4dae-95a6-6cf6a6d617c4 nodeName:}" failed. No retries permitted until 2026-09-10 12:48:19.618993075 +0000 UTC m=+1066.245190628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls") pod "feast-driver-ranking-75d7946bc4-jwxn9" (UID: "8833229f-9604-4dae-95a6-6cf6a6d617c4") : secret "feast-driver-ranking-online-tls" not found Sep 10 12:48:19.625567 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:19.625524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:19.628077 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:19.628057 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") pod \"feast-driver-ranking-75d7946bc4-jwxn9\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:19.752616 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:19.752569 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:20.033364 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:20.033343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9"] Sep 10 12:48:20.035402 ip-10-0-137-89 kubenswrapper[2562]: W0910 12:48:20.035378 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8833229f_9604_4dae_95a6_6cf6a6d617c4.slice/crio-9923cc7bcf1d225dc35122b483ee1ede5f2e3862c40dca0f4c8942e5af058ce9 WatchSource:0}: Error finding container 9923cc7bcf1d225dc35122b483ee1ede5f2e3862c40dca0f4c8942e5af058ce9: Status 404 returned error can't find the container with id 9923cc7bcf1d225dc35122b483ee1ede5f2e3862c40dca0f4c8942e5af058ce9 Sep 10 12:48:20.442563 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:20.442474 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerStarted","Data":"9e8003533784c217b1fe5add3a1ef029545e036597094917e8319ec5c3fea99b"} Sep 10 12:48:20.442563 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:20.442549 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerStarted","Data":"9923cc7bcf1d225dc35122b483ee1ede5f2e3862c40dca0f4c8942e5af058ce9"} Sep 10 12:48:25.467737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:25.467651 2562 generic.go:356] "Generic (PLEG): container finished" podID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerID="9e8003533784c217b1fe5add3a1ef029545e036597094917e8319ec5c3fea99b" exitCode=0 Sep 10 12:48:25.467737 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:25.467701 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerDied","Data":"9e8003533784c217b1fe5add3a1ef029545e036597094917e8319ec5c3fea99b"} Sep 10 12:48:26.473814 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:26.473773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerStarted","Data":"50986a53ea6724d511802322c4a584f7b234c61237411c922fd2b2572859d079"} Sep 10 12:48:32.504332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:32.504298 2562 generic.go:356] "Generic (PLEG): container finished" podID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerID="50986a53ea6724d511802322c4a584f7b234c61237411c922fd2b2572859d079" exitCode=0 Sep 10 12:48:32.504780 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:32.504378 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerDied","Data":"50986a53ea6724d511802322c4a584f7b234c61237411c922fd2b2572859d079"} Sep 10 12:48:33.512672 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:33.512633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerStarted","Data":"34667d6fb749e53333190d4ced48cddbfa6fc10c14ee772e4f6b8e0d8b51571f"} Sep 10 12:48:33.513039 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:33.512712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerStarted","Data":"1d00c343f296936e7c0b588654eb2c2a0cb10c34c6a7b75c678baca6367d5a56"} Sep 10 12:48:33.550189 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:33.550130 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podStartSLOduration=16.550106041 podStartE2EDuration="16.550106041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:33.546448033 +0000 UTC m=+1080.172645609" watchObservedRunningTime="2026-09-10 12:48:33.550106041 +0000 UTC m=+1080.176303617" Sep 10 12:48:34.752829 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:34.752791 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:34.753271 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:34.752859 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:34.754431 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:34.754403 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 10 12:48:34.754546 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:34.754441 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 10 12:48:37.753454 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:37.753418 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 10 12:48:37.753837 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:37.753411 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 10 12:48:39.753369 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:39.753334 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:39.753758 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:39.753394 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:40.752994 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:40.752950 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 10 12:48:40.759137 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:40.759107 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:40.762462 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:40.762418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:43.752915 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:43.752863 2562 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.30:6571: connect: connection refused" Sep 10 12:48:46.753807 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:46.753780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:48:46.754544 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:48:46.754527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:50:05.898683 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:05.898642 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj"] Sep 10 12:50:05.899404 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:05.898969 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" containerID="cri-o://a54f0adef70fc02840c43bbc60e587f896c7469dc516b926166a2881d0ecbde9" gracePeriod=30 Sep 10 12:50:05.899404 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:05.898963 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" containerID="cri-o://aec6b02b9c3884fbe402982683148c685ef39901a355664e115e03458fe50a06" gracePeriod=30 Sep 10 12:50:05.901621 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:05.901356 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9"] Sep 10 12:50:05.901621 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:05.901562 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" containerID="cri-o://1d00c343f296936e7c0b588654eb2c2a0cb10c34c6a7b75c678baca6367d5a56" gracePeriod=30 Sep 10 12:50:05.901812 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:05.901751 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" containerID="cri-o://34667d6fb749e53333190d4ced48cddbfa6fc10c14ee772e4f6b8e0d8b51571f" gracePeriod=30 Sep 10 12:50:06.918056 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:06.917970 2562 generic.go:356] "Generic (PLEG): container finished" podID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerID="a54f0adef70fc02840c43bbc60e587f896c7469dc516b926166a2881d0ecbde9" exitCode=0 Sep 10 12:50:06.918056 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:06.918046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerDied","Data":"a54f0adef70fc02840c43bbc60e587f896c7469dc516b926166a2881d0ecbde9"} Sep 10 12:50:06.920227 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:06.920209 2562 generic.go:356] "Generic (PLEG): container finished" podID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerID="34667d6fb749e53333190d4ced48cddbfa6fc10c14ee772e4f6b8e0d8b51571f" exitCode=0 Sep 10 12:50:06.920324 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:06.920266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerDied","Data":"34667d6fb749e53333190d4ced48cddbfa6fc10c14ee772e4f6b8e0d8b51571f"} Sep 10 12:50:08.215564 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.215524 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-dv7lf"] Sep 10 12:50:08.216091 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.215772 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" podUID="b75529bb-7d83-43a8-9111-a418e792ee5f" containerName="postgres" containerID="cri-o://1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f" gracePeriod=30 Sep 10 12:50:08.453463 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.453439 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:50:08.582697 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.582664 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b75529bb-7d83-43a8-9111-a418e792ee5f-postgresdata\") pod \"b75529bb-7d83-43a8-9111-a418e792ee5f\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " Sep 10 12:50:08.582847 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.582713 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99bnt\" (UniqueName: \"kubernetes.io/projected/b75529bb-7d83-43a8-9111-a418e792ee5f-kube-api-access-99bnt\") pod \"b75529bb-7d83-43a8-9111-a418e792ee5f\" (UID: \"b75529bb-7d83-43a8-9111-a418e792ee5f\") " Sep 10 12:50:08.584920 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.584890 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75529bb-7d83-43a8-9111-a418e792ee5f-kube-api-access-99bnt" (OuterVolumeSpecName: "kube-api-access-99bnt") pod "b75529bb-7d83-43a8-9111-a418e792ee5f" (UID: "b75529bb-7d83-43a8-9111-a418e792ee5f"). InnerVolumeSpecName "kube-api-access-99bnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:50:08.612085 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.612055 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75529bb-7d83-43a8-9111-a418e792ee5f-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "b75529bb-7d83-43a8-9111-a418e792ee5f" (UID: "b75529bb-7d83-43a8-9111-a418e792ee5f"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:50:08.683243 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.683218 2562 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b75529bb-7d83-43a8-9111-a418e792ee5f-postgresdata\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:08.683243 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.683243 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99bnt\" (UniqueName: \"kubernetes.io/projected/b75529bb-7d83-43a8-9111-a418e792ee5f-kube-api-access-99bnt\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:08.928874 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.928787 2562 generic.go:356] "Generic (PLEG): container finished" podID="b75529bb-7d83-43a8-9111-a418e792ee5f" containerID="1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f" exitCode=0 Sep 10 12:50:08.928874 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.928842 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" Sep 10 12:50:08.929090 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.928878 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" event={"ID":"b75529bb-7d83-43a8-9111-a418e792ee5f","Type":"ContainerDied","Data":"1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f"} Sep 10 12:50:08.929090 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.928924 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-dv7lf" event={"ID":"b75529bb-7d83-43a8-9111-a418e792ee5f","Type":"ContainerDied","Data":"6553af0c6ad2ba7a75b079541f286dca5e111c14564cf5b8b08f91e0a0cb5bc6"} Sep 10 12:50:08.929090 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.928944 2562 scope.go:117] "RemoveContainer" containerID="1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f" Sep 10 12:50:08.937902 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.937885 2562 scope.go:117] "RemoveContainer" containerID="1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f" Sep 10 12:50:08.938188 ip-10-0-137-89 kubenswrapper[2562]: E0910 12:50:08.938170 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f\": container with ID starting with 1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f not found: ID does not exist" containerID="1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f" Sep 10 12:50:08.938230 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.938197 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f"} err="failed to get container status \"1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f\": rpc error: code = NotFound desc = could not find container \"1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f\": container with ID starting with 1a56ea1d4ab7b9e663409c62b8bc9998b0102bdbfb146590dfc6158ab797495f not found: ID does not exist" Sep 10 12:50:08.951732 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.951712 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-dv7lf"] Sep 10 12:50:08.957044 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:08.957026 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-dv7lf"] Sep 10 12:50:09.837799 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:09.837769 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75529bb-7d83-43a8-9111-a418e792ee5f" path="/var/lib/kubelet/pods/b75529bb-7d83-43a8-9111-a418e792ee5f/volumes" Sep 10 12:50:10.454215 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:10.454181 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 10 12:50:10.760455 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:10.760426 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 10 12:50:20.454289 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:20.454248 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 10 12:50:20.760085 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:20.760048 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 10 12:50:30.453359 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:30.453317 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 10 12:50:30.453863 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:30.453460 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:50:30.760448 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:30.760410 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" probeResult="failure" output="Get \"https://10.134.0.30:6567/health\": dial tcp 10.134.0.30:6567: connect: connection refused" Sep 10 12:50:30.760632 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:30.760549 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:50:36.042261 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.042235 2562 generic.go:356] "Generic (PLEG): container finished" podID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerID="aec6b02b9c3884fbe402982683148c685ef39901a355664e115e03458fe50a06" exitCode=137 Sep 10 12:50:36.042565 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.042310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerDied","Data":"aec6b02b9c3884fbe402982683148c685ef39901a355664e115e03458fe50a06"} Sep 10 12:50:36.044806 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.044784 2562 generic.go:356] "Generic (PLEG): container finished" podID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerID="1d00c343f296936e7c0b588654eb2c2a0cb10c34c6a7b75c678baca6367d5a56" exitCode=137 Sep 10 12:50:36.044910 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.044820 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerDied","Data":"1d00c343f296936e7c0b588654eb2c2a0cb10c34c6a7b75c678baca6367d5a56"} Sep 10 12:50:36.572792 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.572764 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:50:36.575668 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.575649 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:50:36.580752 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580734 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-registry-tls\") pod \"8833229f-9604-4dae-95a6-6cf6a6d617c4\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " Sep 10 12:50:36.580838 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580778 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-online-tls\") pod \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " Sep 10 12:50:36.580838 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8833229f-9604-4dae-95a6-6cf6a6d617c4-feast-data\") pod \"8833229f-9604-4dae-95a6-6cf6a6d617c4\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " Sep 10 12:50:36.580913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580849 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf9sw\" (UniqueName: \"kubernetes.io/projected/bc55cae3-f3ca-4e44-81d9-32e13e172a56-kube-api-access-kf9sw\") pod \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " Sep 10 12:50:36.580913 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") pod \"8833229f-9604-4dae-95a6-6cf6a6d617c4\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " Sep 10 12:50:36.581010 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580933 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-registry-tls\") pod \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " Sep 10 12:50:36.581010 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580963 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc55cae3-f3ca-4e44-81d9-32e13e172a56-feast-data\") pod \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\" (UID: \"bc55cae3-f3ca-4e44-81d9-32e13e172a56\") " Sep 10 12:50:36.581010 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.580997 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xp9v\" (UniqueName: \"kubernetes.io/projected/8833229f-9604-4dae-95a6-6cf6a6d617c4-kube-api-access-4xp9v\") pod \"8833229f-9604-4dae-95a6-6cf6a6d617c4\" (UID: \"8833229f-9604-4dae-95a6-6cf6a6d617c4\") " Sep 10 12:50:36.581744 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.581715 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8833229f-9604-4dae-95a6-6cf6a6d617c4-feast-data" (OuterVolumeSpecName: "feast-data") pod "8833229f-9604-4dae-95a6-6cf6a6d617c4" (UID: "8833229f-9604-4dae-95a6-6cf6a6d617c4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:50:36.583749 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.583725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-online-tls" (OuterVolumeSpecName: "online-tls") pod "bc55cae3-f3ca-4e44-81d9-32e13e172a56" (UID: "bc55cae3-f3ca-4e44-81d9-32e13e172a56"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:50:36.583844 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.583755 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls" (OuterVolumeSpecName: "online-tls") pod "8833229f-9604-4dae-95a6-6cf6a6d617c4" (UID: "8833229f-9604-4dae-95a6-6cf6a6d617c4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:50:36.583844 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.583777 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8833229f-9604-4dae-95a6-6cf6a6d617c4" (UID: "8833229f-9604-4dae-95a6-6cf6a6d617c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:50:36.583844 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.583773 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc55cae3-f3ca-4e44-81d9-32e13e172a56-kube-api-access-kf9sw" (OuterVolumeSpecName: "kube-api-access-kf9sw") pod "bc55cae3-f3ca-4e44-81d9-32e13e172a56" (UID: "bc55cae3-f3ca-4e44-81d9-32e13e172a56"). InnerVolumeSpecName "kube-api-access-kf9sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:50:36.583979 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.583837 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bc55cae3-f3ca-4e44-81d9-32e13e172a56" (UID: "bc55cae3-f3ca-4e44-81d9-32e13e172a56"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 12:50:36.583979 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.583918 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8833229f-9604-4dae-95a6-6cf6a6d617c4-kube-api-access-4xp9v" (OuterVolumeSpecName: "kube-api-access-4xp9v") pod "8833229f-9604-4dae-95a6-6cf6a6d617c4" (UID: "8833229f-9604-4dae-95a6-6cf6a6d617c4"). InnerVolumeSpecName "kube-api-access-4xp9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 12:50:36.590304 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.590281 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc55cae3-f3ca-4e44-81d9-32e13e172a56-feast-data" (OuterVolumeSpecName: "feast-data") pod "bc55cae3-f3ca-4e44-81d9-32e13e172a56" (UID: "bc55cae3-f3ca-4e44-81d9-32e13e172a56"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682375 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xp9v\" (UniqueName: \"kubernetes.io/projected/8833229f-9604-4dae-95a6-6cf6a6d617c4-kube-api-access-4xp9v\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682403 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682416 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-online-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682426 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/8833229f-9604-4dae-95a6-6cf6a6d617c4-feast-data\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682437 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9sw\" (UniqueName: \"kubernetes.io/projected/bc55cae3-f3ca-4e44-81d9-32e13e172a56-kube-api-access-kf9sw\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682449 2562 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/8833229f-9604-4dae-95a6-6cf6a6d617c4-online-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682458 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682460 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bc55cae3-f3ca-4e44-81d9-32e13e172a56-registry-tls\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:36.682821 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:36.682471 2562 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bc55cae3-f3ca-4e44-81d9-32e13e172a56-feast-data\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 12:50:37.050872 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.050835 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" event={"ID":"bc55cae3-f3ca-4e44-81d9-32e13e172a56","Type":"ContainerDied","Data":"16d0fcc29c2c3eedfdc2c60c749152033d2c36ca60e5f32dde2dbb60eb3fa6c5"} Sep 10 12:50:37.050872 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.050856 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj" Sep 10 12:50:37.051315 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.050919 2562 scope.go:117] "RemoveContainer" containerID="a54f0adef70fc02840c43bbc60e587f896c7469dc516b926166a2881d0ecbde9" Sep 10 12:50:37.053332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.053315 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" Sep 10 12:50:37.053332 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.053319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9" event={"ID":"8833229f-9604-4dae-95a6-6cf6a6d617c4","Type":"ContainerDied","Data":"9923cc7bcf1d225dc35122b483ee1ede5f2e3862c40dca0f4c8942e5af058ce9"} Sep 10 12:50:37.059946 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.059856 2562 scope.go:117] "RemoveContainer" containerID="aec6b02b9c3884fbe402982683148c685ef39901a355664e115e03458fe50a06" Sep 10 12:50:37.067916 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.067901 2562 scope.go:117] "RemoveContainer" containerID="9df417805f03e396f132b1d9b7a1bb0efd8e5880ebc347666dd5a2de53ec3dd1" Sep 10 12:50:37.075725 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.075708 2562 scope.go:117] "RemoveContainer" containerID="0ac4ced518e9e5039637a727b39c8ff3c317094b30011557ac4f9cb48b730227" Sep 10 12:50:37.076080 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.076063 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj"] Sep 10 12:50:37.080662 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.080642 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-7ff84d6969-rhghj"] Sep 10 12:50:37.083358 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.083342 2562 scope.go:117] "RemoveContainer" containerID="34667d6fb749e53333190d4ced48cddbfa6fc10c14ee772e4f6b8e0d8b51571f" Sep 10 12:50:37.090616 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.090601 2562 scope.go:117] "RemoveContainer" containerID="1d00c343f296936e7c0b588654eb2c2a0cb10c34c6a7b75c678baca6367d5a56" Sep 10 12:50:37.094434 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.094413 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9"] Sep 10 12:50:37.098018 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.098001 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-75d7946bc4-jwxn9"] Sep 10 12:50:37.099216 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.099199 2562 scope.go:117] "RemoveContainer" containerID="50986a53ea6724d511802322c4a584f7b234c61237411c922fd2b2572859d079" Sep 10 12:50:37.106025 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.106010 2562 scope.go:117] "RemoveContainer" containerID="9e8003533784c217b1fe5add3a1ef029545e036597094917e8319ec5c3fea99b" Sep 10 12:50:37.838577 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.838526 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" path="/var/lib/kubelet/pods/8833229f-9604-4dae-95a6-6cf6a6d617c4/volumes" Sep 10 12:50:37.839157 ip-10-0-137-89 kubenswrapper[2562]: I0910 12:50:37.839131 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" path="/var/lib/kubelet/pods/bc55cae3-f3ca-4e44-81d9-32e13e172a56/volumes" Sep 10 13:02:35.818722 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.818687 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xbmwj/must-gather-df2g4"] Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.818984 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="feast-init" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.818994 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="feast-init" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819003 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="feast-apply" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819009 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="feast-apply" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819018 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="feast-init" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819023 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="feast-init" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819034 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819039 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819045 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819050 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819061 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819065 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819074 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819078 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819084 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75529bb-7d83-43a8-9111-a418e792ee5f" containerName="postgres" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819089 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75529bb-7d83-43a8-9111-a418e792ee5f" containerName="postgres" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819097 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="feast-apply" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819101 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="feast-apply" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819144 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="online" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819153 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc55cae3-f3ca-4e44-81d9-32e13e172a56" containerName="registry" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819161 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="registry" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819168 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75529bb-7d83-43a8-9111-a418e792ee5f" containerName="postgres" Sep 10 13:02:35.819205 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.819175 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8833229f-9604-4dae-95a6-6cf6a6d617c4" containerName="online" Sep 10 13:02:35.826135 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.826111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:35.828946 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.828925 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xbmwj\"/\"default-dockercfg-glndv\"" Sep 10 13:02:35.829075 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.828976 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbmwj\"/\"openshift-service-ca.crt\"" Sep 10 13:02:35.829977 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.829961 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbmwj\"/\"kube-root-ca.crt\"" Sep 10 13:02:35.839145 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.839123 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbmwj/must-gather-df2g4"] Sep 10 13:02:35.932950 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.932919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpgv2\" (UniqueName: \"kubernetes.io/projected/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-kube-api-access-wpgv2\") pod \"must-gather-df2g4\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:35.933104 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:35.932956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-must-gather-output\") pod \"must-gather-df2g4\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:36.033496 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.033467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpgv2\" (UniqueName: \"kubernetes.io/projected/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-kube-api-access-wpgv2\") pod \"must-gather-df2g4\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:36.033658 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.033514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-must-gather-output\") pod \"must-gather-df2g4\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:36.033905 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.033877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-must-gather-output\") pod \"must-gather-df2g4\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:36.046228 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.046198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpgv2\" (UniqueName: \"kubernetes.io/projected/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-kube-api-access-wpgv2\") pod \"must-gather-df2g4\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:36.141648 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.141570 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:36.421614 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.421569 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbmwj/must-gather-df2g4"] Sep 10 13:02:36.424251 ip-10-0-137-89 kubenswrapper[2562]: W0910 13:02:36.424224 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b8b088a_a25b_42fa_a3ec_6f73aa65c896.slice/crio-64af6f7e38395459f7fbe2c17ff59ecb60a5f452070343b0ecdc8c0295e7d1ed WatchSource:0}: Error finding container 64af6f7e38395459f7fbe2c17ff59ecb60a5f452070343b0ecdc8c0295e7d1ed: Status 404 returned error can't find the container with id 64af6f7e38395459f7fbe2c17ff59ecb60a5f452070343b0ecdc8c0295e7d1ed Sep 10 13:02:36.426043 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.426026 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 13:02:36.888485 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:36.888451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbmwj/must-gather-df2g4" event={"ID":"9b8b088a-a25b-42fa-a3ec-6f73aa65c896","Type":"ContainerStarted","Data":"64af6f7e38395459f7fbe2c17ff59ecb60a5f452070343b0ecdc8c0295e7d1ed"} Sep 10 13:02:43.923183 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:43.923129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbmwj/must-gather-df2g4" event={"ID":"9b8b088a-a25b-42fa-a3ec-6f73aa65c896","Type":"ContainerStarted","Data":"63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d"} Sep 10 13:02:43.923600 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:43.923192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbmwj/must-gather-df2g4" event={"ID":"9b8b088a-a25b-42fa-a3ec-6f73aa65c896","Type":"ContainerStarted","Data":"4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab"} Sep 10 13:02:43.951779 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:43.951735 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xbmwj/must-gather-df2g4" podStartSLOduration=2.058800738 podStartE2EDuration="8.951720933s" totalImagesPullingTime="6.892920195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:02:43.951539477 +0000 UTC m=+1930.577737052" watchObservedRunningTime="2026-09-10 13:02:43.951720933 +0000 UTC m=+1930.577918507" Sep 10 13:02:52.962359 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:52.962323 2562 generic.go:356] "Generic (PLEG): container finished" podID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerID="4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab" exitCode=0 Sep 10 13:02:52.962865 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:52.962390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbmwj/must-gather-df2g4" event={"ID":"9b8b088a-a25b-42fa-a3ec-6f73aa65c896","Type":"ContainerDied","Data":"4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab"} Sep 10 13:02:52.962865 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:52.962750 2562 scope.go:117] "RemoveContainer" containerID="4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab" Sep 10 13:02:53.836303 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:53.836256 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xbmwj_must-gather-df2g4_9b8b088a-a25b-42fa-a3ec-6f73aa65c896/gather/0.log" Sep 10 13:02:57.437060 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:57.437019 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5s256_46be07f1-b76e-451e-a75e-0ac2af42c40a/global-pull-secret-syncer/0.log" Sep 10 13:02:57.749391 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:57.749312 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mmnx4_f0fcf238-0942-47a0-b348-896e5049a982/konnectivity-agent/0.log" Sep 10 13:02:57.820665 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:57.820639 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-89.ec2.internal_05dc69e2d25cad8ef7303fdcd6d064db/haproxy/0.log" Sep 10 13:02:59.239251 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.239215 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-xbmwj/must-gather-df2g4"] Sep 10 13:02:59.239684 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.239397 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-xbmwj/must-gather-df2g4" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="copy" containerID="cri-o://63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d" gracePeriod=2 Sep 10 13:02:59.244484 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.244450 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-xbmwj/must-gather-df2g4"] Sep 10 13:02:59.469409 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.469388 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xbmwj_must-gather-df2g4_9b8b088a-a25b-42fa-a3ec-6f73aa65c896/copy/0.log" Sep 10 13:02:59.471056 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.471037 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:59.473532 ip-10-0-137-89 kubenswrapper[2562]: E0910 13:02:59.473509 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-df2g4\" is forbidden: User \"system:node:ip-10-0-137-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-xbmwj\": no relationship found between node 'ip-10-0-137-89.ec2.internal' and this object" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:59.521423 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.521403 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-must-gather-output\") pod \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " Sep 10 13:02:59.521501 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.521468 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpgv2\" (UniqueName: \"kubernetes.io/projected/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-kube-api-access-wpgv2\") pod \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\" (UID: \"9b8b088a-a25b-42fa-a3ec-6f73aa65c896\") " Sep 10 13:02:59.521775 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.521750 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "9b8b088a-a25b-42fa-a3ec-6f73aa65c896" (UID: "9b8b088a-a25b-42fa-a3ec-6f73aa65c896"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 13:02:59.523845 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.523822 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-kube-api-access-wpgv2" (OuterVolumeSpecName: "kube-api-access-wpgv2") pod "9b8b088a-a25b-42fa-a3ec-6f73aa65c896" (UID: "9b8b088a-a25b-42fa-a3ec-6f73aa65c896"). InnerVolumeSpecName "kube-api-access-wpgv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 13:02:59.622272 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.622247 2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-must-gather-output\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 13:02:59.622272 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.622268 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpgv2\" (UniqueName: \"kubernetes.io/projected/9b8b088a-a25b-42fa-a3ec-6f73aa65c896-kube-api-access-wpgv2\") on node \"ip-10-0-137-89.ec2.internal\" DevicePath \"\"" Sep 10 13:02:59.837962 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.837903 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" path="/var/lib/kubelet/pods/9b8b088a-a25b-42fa-a3ec-6f73aa65c896/volumes" Sep 10 13:02:59.993633 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.993605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-xbmwj_must-gather-df2g4_9b8b088a-a25b-42fa-a3ec-6f73aa65c896/copy/0.log" Sep 10 13:02:59.993953 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.993933 2562 generic.go:356] "Generic (PLEG): container finished" podID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerID="63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d" exitCode=143 Sep 10 13:02:59.994038 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.993993 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbmwj/must-gather-df2g4" Sep 10 13:02:59.994143 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:02:59.994036 2562 scope.go:117] "RemoveContainer" containerID="63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d" Sep 10 13:03:00.001535 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:00.001519 2562 scope.go:117] "RemoveContainer" containerID="4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab" Sep 10 13:03:00.010941 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:00.010924 2562 scope.go:117] "RemoveContainer" containerID="63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d" Sep 10 13:03:00.011206 ip-10-0-137-89 kubenswrapper[2562]: E0910 13:03:00.011186 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d\": container with ID starting with 63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d not found: ID does not exist" containerID="63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d" Sep 10 13:03:00.011295 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:00.011211 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d"} err="failed to get container status \"63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d\": rpc error: code = NotFound desc = could not find container \"63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d\": container with ID starting with 63c8dd275b3c8c000bc75dfeeec3d859c650fc7bdd0914dfc1325c806d01a85d not found: ID does not exist" Sep 10 13:03:00.011295 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:00.011229 2562 scope.go:117] "RemoveContainer" containerID="4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab" Sep 10 13:03:00.011468 ip-10-0-137-89 kubenswrapper[2562]: E0910 13:03:00.011452 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab\": container with ID starting with 4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab not found: ID does not exist" containerID="4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab" Sep 10 13:03:00.011518 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:00.011473 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab"} err="failed to get container status \"4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab\": rpc error: code = NotFound desc = could not find container \"4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab\": container with ID starting with 4ab49df23c95e3560d1ac9233f7085b118d7037a6c6078fbcd1bbd76eb413cab not found: ID does not exist" Sep 10 13:03:02.103495 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.103412 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-w9prz_fbe55502-a5c4-40a4-88f8-16e462c1e3b4/kube-state-metrics/0.log" Sep 10 13:03:02.146062 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.146022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-w9prz_fbe55502-a5c4-40a4-88f8-16e462c1e3b4/kube-rbac-proxy-main/0.log" Sep 10 13:03:02.184427 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.184394 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-w9prz_fbe55502-a5c4-40a4-88f8-16e462c1e3b4/kube-rbac-proxy-self/0.log" Sep 10 13:03:02.241793 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.241774 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cd5c8d54b-wprd5_65ace7c6-1680-46b5-8cc3-40b231626375/metrics-server/0.log" Sep 10 13:03:02.328310 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.328289 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wxrt_c24c7cd2-ab0f-43be-935f-861896a14c1b/node-exporter/0.log" Sep 10 13:03:02.375204 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.375143 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wxrt_c24c7cd2-ab0f-43be-935f-861896a14c1b/kube-rbac-proxy/0.log" Sep 10 13:03:02.439486 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.439469 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wxrt_c24c7cd2-ab0f-43be-935f-861896a14c1b/init-textfile/0.log" Sep 10 13:03:02.980349 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:02.980288 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/prometheus/0.log" Sep 10 13:03:03.022923 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.022897 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/config-reloader/0.log" Sep 10 13:03:03.065115 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.065095 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/thanos-sidecar/0.log" Sep 10 13:03:03.100307 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.100287 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/kube-rbac-proxy-web/0.log" Sep 10 13:03:03.134302 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.134281 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/kube-rbac-proxy/0.log" Sep 10 13:03:03.172997 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.172979 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/kube-rbac-proxy-thanos/0.log" Sep 10 13:03:03.214815 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.214797 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_15f309fb-6ae8-423b-bd84-20a30985db24/init-config-reloader/0.log" Sep 10 13:03:03.333727 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.333708 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-vn7np_2f156b43-9250-40af-8193-d3300a188afc/prometheus-operator-admission-webhook/0.log" Sep 10 13:03:03.517385 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.517367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766c99fbbd-7fb2c_6254c17b-eb8d-4c15-a848-14d0f8d2e078/thanos-query/0.log" Sep 10 13:03:03.544520 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.544501 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766c99fbbd-7fb2c_6254c17b-eb8d-4c15-a848-14d0f8d2e078/kube-rbac-proxy-web/0.log" Sep 10 13:03:03.574781 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.574764 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766c99fbbd-7fb2c_6254c17b-eb8d-4c15-a848-14d0f8d2e078/kube-rbac-proxy/0.log" Sep 10 13:03:03.600683 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.600631 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766c99fbbd-7fb2c_6254c17b-eb8d-4c15-a848-14d0f8d2e078/prom-label-proxy/0.log" Sep 10 13:03:03.646381 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.646353 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766c99fbbd-7fb2c_6254c17b-eb8d-4c15-a848-14d0f8d2e078/kube-rbac-proxy-rules/0.log" Sep 10 13:03:03.674115 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:03.674096 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766c99fbbd-7fb2c_6254c17b-eb8d-4c15-a848-14d0f8d2e078/kube-rbac-proxy-metrics/0.log" Sep 10 13:03:04.871504 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.871464 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn"] Sep 10 13:03:04.872020 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.872001 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="gather" Sep 10 13:03:04.872091 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.872023 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="gather" Sep 10 13:03:04.872091 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.872057 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="copy" Sep 10 13:03:04.872091 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.872067 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="copy" Sep 10 13:03:04.872297 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.872145 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="gather" Sep 10 13:03:04.872297 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.872156 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b8b088a-a25b-42fa-a3ec-6f73aa65c896" containerName="copy" Sep 10 13:03:04.876013 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.875990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:04.878476 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.878449 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nd6nf\"/\"kube-root-ca.crt\"" Sep 10 13:03:04.879300 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.879276 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nd6nf\"/\"openshift-service-ca.crt\"" Sep 10 13:03:04.879389 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.879284 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nd6nf\"/\"default-dockercfg-9f665\"" Sep 10 13:03:04.893340 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.893317 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn"] Sep 10 13:03:04.963052 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.963023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-proc\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:04.963175 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.963064 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgtb\" (UniqueName: \"kubernetes.io/projected/9e374294-c0fa-48c0-9609-68c0fe7231ec-kube-api-access-jfgtb\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:04.963175 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.963095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-podres\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:04.963175 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.963120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-lib-modules\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:04.963175 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:04.963151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-sys\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.063727 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.063700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-proc\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.063848 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.063742 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgtb\" (UniqueName: \"kubernetes.io/projected/9e374294-c0fa-48c0-9609-68c0fe7231ec-kube-api-access-jfgtb\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.063848 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.063813 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-proc\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.063968 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.063867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-podres\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.063968 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.063901 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-lib-modules\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.064068 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.064008 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-podres\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.064068 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.064008 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-sys\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.064068 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.064038 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-sys\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.064068 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.064004 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e374294-c0fa-48c0-9609-68c0fe7231ec-lib-modules\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.073968 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.073946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgtb\" (UniqueName: \"kubernetes.io/projected/9e374294-c0fa-48c0-9609-68c0fe7231ec-kube-api-access-jfgtb\") pod \"perf-node-gather-daemonset-bzcjn\" (UID: \"9e374294-c0fa-48c0-9609-68c0fe7231ec\") " pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.186474 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.186420 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:05.460424 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.460402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn"] Sep 10 13:03:05.463301 ip-10-0-137-89 kubenswrapper[2562]: W0910 13:03:05.463270 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e374294_c0fa_48c0_9609_68c0fe7231ec.slice/crio-30048116a981eacf6e8a1de0342b891f0d6804a52dc73a3274c702ae3c559df0 WatchSource:0}: Error finding container 30048116a981eacf6e8a1de0342b891f0d6804a52dc73a3274c702ae3c559df0: Status 404 returned error can't find the container with id 30048116a981eacf6e8a1de0342b891f0d6804a52dc73a3274c702ae3c559df0 Sep 10 13:03:05.689626 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:05.689599 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-vhhs9_5eb3e666-7520-40e7-b6d1-962fab87d027/download-server/0.log" Sep 10 13:03:06.022438 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:06.022400 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" event={"ID":"9e374294-c0fa-48c0-9609-68c0fe7231ec","Type":"ContainerStarted","Data":"3d00baadaaa6b46ab06ba97b2f3a91c66e25338fc67fcc9da444214b8b3a695f"} Sep 10 13:03:06.022845 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:06.022452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" event={"ID":"9e374294-c0fa-48c0-9609-68c0fe7231ec","Type":"ContainerStarted","Data":"30048116a981eacf6e8a1de0342b891f0d6804a52dc73a3274c702ae3c559df0"} Sep 10 13:03:06.022845 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:06.022485 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:06.041337 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:06.041293 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" podStartSLOduration=2.041278751 podStartE2EDuration="2.041278751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:03:06.04018997 +0000 UTC m=+1952.666387557" watchObservedRunningTime="2026-09-10 13:03:06.041278751 +0000 UTC m=+1952.667476326" Sep 10 13:03:06.164261 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:06.164237 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-lj58t_e6855898-d009-4ae4-a5b1-b44df64edb60/volume-data-source-validator/0.log" Sep 10 13:03:07.173412 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:07.173381 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjtjh_7d5f17aa-ec97-43cc-b367-e66e57ff9503/dns/0.log" Sep 10 13:03:07.209803 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:07.209782 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jjtjh_7d5f17aa-ec97-43cc-b367-e66e57ff9503/kube-rbac-proxy/0.log" Sep 10 13:03:07.272066 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:07.272041 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5kcqc_57937ad4-a410-4961-8ba1-e72addf6c5a3/dns-node-resolver/0.log" Sep 10 13:03:07.863142 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:07.863100 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zz8th_9a6dfcda-b4c5-41c0-9c31-53995fb29b8c/node-ca/0.log" Sep 10 13:03:09.076542 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:09.076516 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-chwd4_7c311b6e-3312-4db0-9623-7e4b662e08ca/serve-healthcheck-canary/0.log" Sep 10 13:03:09.569599 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:09.569544 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpxpr_3b03053e-fdd6-4256-8300-cfcd0c67e5f8/kube-rbac-proxy/0.log" Sep 10 13:03:09.593048 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:09.593027 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpxpr_3b03053e-fdd6-4256-8300-cfcd0c67e5f8/exporter/0.log" Sep 10 13:03:09.622044 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:09.622022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mpxpr_3b03053e-fdd6-4256-8300-cfcd0c67e5f8/extractor/0.log" Sep 10 13:03:12.036813 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:12.036780 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nd6nf/perf-node-gather-daemonset-bzcjn" Sep 10 13:03:14.726131 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:14.726056 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-svhwc_a0b64a56-a956-41c8-a0df-24209006a2ea/migrator/0.log" Sep 10 13:03:14.754057 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:14.754034 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-svhwc_a0b64a56-a956-41c8-a0df-24209006a2ea/graceful-termination/0.log" Sep 10 13:03:15.107573 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:15.107544 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rtqmv_88e683cb-1f68-45b1-bcd9-a85bb66c9e1c/kube-storage-version-migrator-operator/1.log" Sep 10 13:03:15.108297 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:15.108279 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-rtqmv_88e683cb-1f68-45b1-bcd9-a85bb66c9e1c/kube-storage-version-migrator-operator/0.log" Sep 10 13:03:16.426686 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.426658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/kube-multus-additional-cni-plugins/0.log" Sep 10 13:03:16.449046 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.449026 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/egress-router-binary-copy/0.log" Sep 10 13:03:16.474481 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.474464 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/cni-plugins/0.log" Sep 10 13:03:16.494598 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.494572 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/bond-cni-plugin/0.log" Sep 10 13:03:16.515981 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.515962 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/routeoverride-cni/0.log" Sep 10 13:03:16.539682 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.539667 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/whereabouts-cni-bincopy/0.log" Sep 10 13:03:16.562456 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.562435 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vkghf_fd98d133-19c5-416b-ba3d-ef46bf070e48/whereabouts-cni/0.log" Sep 10 13:03:16.626471 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.626415 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jhrk2_397fbfb0-a6e3-4df2-9be2-b08d939993b4/kube-multus/0.log" Sep 10 13:03:16.676056 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.676035 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6s69q_3c49d065-9a7d-4b66-b420-0bd16b7dfbae/network-metrics-daemon/0.log" Sep 10 13:03:16.696809 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:16.696786 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6s69q_3c49d065-9a7d-4b66-b420-0bd16b7dfbae/kube-rbac-proxy/0.log" Sep 10 13:03:18.334503 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.334475 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/ovn-controller/0.log" Sep 10 13:03:18.366633 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.366607 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/ovn-acl-logging/0.log" Sep 10 13:03:18.396843 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.396814 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/kube-rbac-proxy-node/0.log" Sep 10 13:03:18.425129 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.425110 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 13:03:18.450780 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.450763 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/northd/0.log" Sep 10 13:03:18.476768 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.476745 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/nbdb/0.log" Sep 10 13:03:18.506676 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.506657 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/sbdb/0.log" Sep 10 13:03:18.550838 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:18.550819 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t5558_17566da3-e5fb-45ca-891a-e1ec4796759a/ovnkube-controller/0.log" Sep 10 13:03:19.754474 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:19.754442 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-twpc8_1110380b-db68-445a-9601-1f6dedd8c1c9/network-check-target-container/0.log" Sep 10 13:03:20.849029 ip-10-0-137-89 kubenswrapper[2562]: I0910 13:03:20.848998 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-7qq24_ba58ecfe-97d8-4d2a-a596-33d5e5c80752/iptables-alerter/0.log"