Sep 16 11:53:07.470702 ip-10-0-140-32 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 11:53:07.470714 ip-10-0-140-32 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 11:53:07.470721 ip-10-0-140-32 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 11:53:07.470992 ip-10-0-140-32 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 11:53:17.680135 ip-10-0-140-32 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 11:53:17.680150 ip-10-0-140-32 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 236d288fd5584cc3b2928e820739fb3f -- Sep 16 11:55:14.101831 ip-10-0-140-32 systemd[1]: Starting Kubernetes Kubelet... Sep 16 11:55:14.494923 ip-10-0-140-32 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 11:55:14.494923 ip-10-0-140-32 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 11:55:14.494923 ip-10-0-140-32 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 11:55:14.494923 ip-10-0-140-32 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 11:55:14.494923 ip-10-0-140-32 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 11:55:14.495914 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.495822 2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 11:55:14.499703 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499682 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 11:55:14.499703 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499702 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499707 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499711 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499714 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499717 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499721 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499724 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499727 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499730 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499732 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499735 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499740 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499744 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499747 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499756 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499760 2551 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499763 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499767 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499769 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 11:55:14.499804 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499772 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499775 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499778 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499780 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499783 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499786 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499788 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499791 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499794 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499796 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499799 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499801 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499804 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499807 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499810 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499815 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499819 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499824 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499827 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 11:55:14.500334 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499830 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499834 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499836 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499839 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499841 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499844 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499847 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499850 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499852 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499856 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499858 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499861 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499864 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499866 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499869 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499873 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499875 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499879 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499882 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 11:55:14.501046 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499884 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499887 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499890 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499892 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499895 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499898 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499900 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499903 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499906 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499908 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499911 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499914 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499916 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499919 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499923 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499925 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499928 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499931 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499934 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499937 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 11:55:14.501602 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499940 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499943 2551 feature_gate.go:328] unrecognized feature gate: Example Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499945 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499948 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499951 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499954 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499957 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499959 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499963 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499965 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499968 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499971 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499973 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499976 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499978 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499981 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499984 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499986 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499989 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499991 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 11:55:14.502088 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.499994 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501845 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501858 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501862 2551 feature_gate.go:328] unrecognized feature gate: Example Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501865 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501868 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501872 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501875 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501878 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501882 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501885 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501888 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501890 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501893 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501896 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501899 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501903 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501906 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501909 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501911 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 11:55:14.502651 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501915 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501918 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501921 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501923 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501926 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501929 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501931 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501934 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501937 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501939 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501942 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501945 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501948 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501952 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501955 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501958 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501961 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501964 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501967 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501970 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 11:55:14.503151 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501972 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501975 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501978 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501981 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501986 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501990 2551 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501993 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501996 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.501999 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502001 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502004 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502007 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502010 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502013 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502016 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502018 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502021 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502024 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502026 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 11:55:14.503696 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502029 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502032 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502034 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502037 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502039 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502042 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502046 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502050 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502053 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502056 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502059 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502062 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502065 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502067 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502071 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502075 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502078 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502080 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502083 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 11:55:14.504165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502085 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502088 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502091 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502094 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502097 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502099 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502104 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502108 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502111 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502113 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502118 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502121 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502123 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502126 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502128 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502131 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502134 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502137 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502140 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 11:55:14.504661 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502143 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502146 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502149 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502578 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502584 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502587 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502591 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502595 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502598 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502601 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502603 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502606 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502609 2551 feature_gate.go:328] unrecognized feature gate: Example Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502612 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502614 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502617 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502620 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502623 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502625 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502629 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 11:55:14.505203 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502632 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502635 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502637 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502640 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502644 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502647 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502649 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502652 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502655 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502658 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502661 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502664 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502667 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502671 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502676 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502679 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502682 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502685 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502688 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 11:55:14.505722 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502690 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502693 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502697 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502700 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502702 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502705 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502708 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502710 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502713 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502716 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502718 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502721 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502723 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502727 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502729 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502732 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502734 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502737 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502741 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502743 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 11:55:14.506188 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502746 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502748 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502751 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502753 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502756 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502758 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502761 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502764 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502767 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502770 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502773 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502777 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502781 2551 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502784 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502787 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502790 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502793 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502797 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502800 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 11:55:14.506704 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502831 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502835 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502839 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502842 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502844 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502847 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502849 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502853 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502856 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502859 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502862 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502865 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502868 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502870 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502873 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502876 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502878 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502881 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502884 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502889 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 11:55:14.507165 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502892 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502895 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502898 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.502900 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.502978 2551 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503023 2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503030 2551 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503035 2551 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503039 2551 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503043 2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503048 2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503052 2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503056 2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503059 2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503063 2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503067 2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503070 2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503074 2551 flags.go:64] FLAG: --cgroup-root="" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503077 2551 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503081 2551 flags.go:64] FLAG: --client-ca-file="" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503085 2551 flags.go:64] FLAG: --cloud-provider="external" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503088 2551 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503093 2551 flags.go:64] FLAG: --cluster-domain="" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503097 2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 11:55:14.507719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503100 2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503104 2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503108 2551 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503112 2551 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503115 2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503118 2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503122 2551 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503127 2551 flags.go:64] FLAG: --contention-profiling="false" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503130 2551 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503134 2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503137 2551 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503140 2551 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503145 2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503148 2551 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503152 2551 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503155 2551 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503159 2551 flags.go:64] FLAG: --enable-server="true" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503162 2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503166 2551 flags.go:64] FLAG: --event-burst="100" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503170 2551 flags.go:64] FLAG: --event-qps="50" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503173 2551 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503176 2551 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503179 2551 flags.go:64] FLAG: --eviction-hard="" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503184 2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 11:55:14.508326 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503187 2551 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503190 2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503193 2551 flags.go:64] FLAG: --eviction-soft="" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503196 2551 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503199 2551 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503203 2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503207 2551 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503210 2551 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503214 2551 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503217 2551 flags.go:64] FLAG: --feature-gates="" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503221 2551 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503224 2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503228 2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503231 2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503235 2551 flags.go:64] FLAG: --healthz-port="10248" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503239 2551 flags.go:64] FLAG: --help="false" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503242 2551 flags.go:64] FLAG: --hostname-override="ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503245 2551 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503248 2551 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503251 2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503254 2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503258 2551 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503261 2551 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 11:55:14.508930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503264 2551 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503267 2551 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503270 2551 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503273 2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503276 2551 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503279 2551 flags.go:64] FLAG: --kube-reserved="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503282 2551 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503285 2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503288 2551 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503291 2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503294 2551 flags.go:64] FLAG: --lock-file="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503297 2551 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503300 2551 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503303 2551 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503309 2551 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503313 2551 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503317 2551 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503320 2551 flags.go:64] FLAG: --logging-format="text" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503322 2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503326 2551 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503329 2551 flags.go:64] FLAG: --manifest-url="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503332 2551 flags.go:64] FLAG: --manifest-url-header="" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503337 2551 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503340 2551 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503345 2551 flags.go:64] FLAG: --max-pods="110" Sep 16 11:55:14.509539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503348 2551 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503352 2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503355 2551 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503358 2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503361 2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503364 2551 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503367 2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503376 2551 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503379 2551 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503382 2551 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503386 2551 flags.go:64] FLAG: --pod-cidr="" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503389 2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503395 2551 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503398 2551 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503401 2551 flags.go:64] FLAG: --pods-per-core="0" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503405 2551 flags.go:64] FLAG: --port="10250" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503408 2551 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503411 2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0598703d1275f6587" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503414 2551 flags.go:64] FLAG: --qos-reserved="" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503417 2551 flags.go:64] FLAG: --read-only-port="10255" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503420 2551 flags.go:64] FLAG: --register-node="true" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503433 2551 flags.go:64] FLAG: --register-with-taints="" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503440 2551 flags.go:64] FLAG: --registry-burst="10" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503444 2551 flags.go:64] FLAG: --registry-qps="5" Sep 16 11:55:14.510150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503447 2551 flags.go:64] FLAG: --reserved-cpus="" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503450 2551 flags.go:64] FLAG: --reserved-memory="" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503454 2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503457 2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503460 2551 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503464 2551 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503467 2551 flags.go:64] FLAG: --runonce="false" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503470 2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503473 2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503476 2551 flags.go:64] FLAG: --seccomp-default="false" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503479 2551 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503482 2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503485 2551 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503489 2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503492 2551 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503510 2551 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503513 2551 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503516 2551 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503519 2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503523 2551 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503526 2551 flags.go:64] FLAG: --system-cgroups="" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503529 2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503534 2551 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503537 2551 flags.go:64] FLAG: --tls-cert-file="" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503540 2551 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 11:55:14.510768 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503545 2551 flags.go:64] FLAG: --tls-min-version="" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503548 2551 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503551 2551 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503554 2551 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503557 2551 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503562 2551 flags.go:64] FLAG: --v="2" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503567 2551 flags.go:64] FLAG: --version="false" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503572 2551 flags.go:64] FLAG: --vmodule="" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503576 2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.503579 2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503709 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503713 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503716 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503719 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503722 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503725 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503728 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503731 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503735 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503737 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503740 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503744 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 11:55:14.511417 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503746 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503749 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503752 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503756 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503760 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503764 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503767 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503769 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503772 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503775 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503778 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503781 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503783 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503786 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503789 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503795 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503800 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503803 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 11:55:14.511991 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503806 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503810 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503813 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503816 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503819 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503822 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503824 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503827 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503830 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503832 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503835 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503838 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503841 2551 feature_gate.go:328] unrecognized feature gate: Example Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503843 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503846 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503848 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503851 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503854 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503857 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503859 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 11:55:14.512446 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503862 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503864 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503867 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503870 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503873 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503875 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503878 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503881 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503883 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503888 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503892 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503895 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503897 2551 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503901 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503904 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503907 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503909 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503912 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503915 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503918 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 11:55:14.512950 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503922 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503926 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503929 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503932 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503935 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503938 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503940 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503943 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503946 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503949 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503952 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503954 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503957 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503960 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503962 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503965 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503968 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503970 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503973 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503976 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 11:55:14.513438 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503979 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503986 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503990 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503993 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503996 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.503998 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.504002 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.504005 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.504007 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.504521 2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.511562 2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.511581 2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.512272 2551 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.512280 2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512345 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 11:55:14.513965 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512350 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512354 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512358 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512361 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512364 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512367 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512371 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512373 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512377 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512380 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512383 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512386 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512389 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512391 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512394 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512397 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512400 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512403 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512407 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512410 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 11:55:14.514347 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512413 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512416 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512419 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512423 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512426 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512429 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512432 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512434 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512437 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512440 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512442 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512445 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512448 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512452 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512454 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512457 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512460 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512463 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512466 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512469 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 11:55:14.514964 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512472 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512474 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512479 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512484 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512488 2551 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512491 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512508 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512512 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512514 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512517 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512520 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512523 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512526 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512530 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512532 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512535 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512539 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512542 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512566 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 11:55:14.515489 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512570 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512574 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512578 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512583 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512586 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512589 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512593 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512597 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512600 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512602 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512605 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512608 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512611 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512613 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512616 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512619 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512622 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512625 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512627 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 11:55:14.515972 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512631 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512633 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512636 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512638 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512641 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512644 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512647 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512650 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512652 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512655 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512658 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512662 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512665 2551 feature_gate.go:328] unrecognized feature gate: Example Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512667 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512670 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512673 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512675 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512678 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512681 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 11:55:14.516450 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512683 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.512688 2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512807 2551 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512812 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512816 2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512819 2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512822 2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512825 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512828 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512831 2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512834 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512837 2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512839 2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512842 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512844 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 11:55:14.516928 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512847 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512851 2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512856 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512858 2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512861 2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512864 2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512867 2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512869 2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512872 2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512875 2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512881 2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512883 2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512886 2551 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512889 2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512891 2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512894 2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512897 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512899 2551 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512902 2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 11:55:14.517311 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512905 2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512907 2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512910 2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512912 2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512915 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512918 2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512921 2551 feature_gate.go:328] unrecognized feature gate: Example Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512924 2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512927 2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512929 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512932 2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512935 2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512937 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512940 2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512943 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512945 2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512948 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512951 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512954 2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512956 2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 11:55:14.517808 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512959 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512962 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512965 2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512968 2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512971 2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512974 2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512977 2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512980 2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512982 2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512985 2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512988 2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512991 2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512993 2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512996 2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.512998 2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513001 2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513004 2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513006 2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513012 2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513014 2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 11:55:14.518319 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513017 2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513020 2551 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513024 2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513027 2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513031 2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513034 2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513038 2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513042 2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513045 2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513048 2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513052 2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513055 2551 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513058 2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513061 2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513064 2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513067 2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513070 2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513073 2551 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513076 2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 11:55:14.518838 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513079 2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513082 2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513084 2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513087 2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513090 2551 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513093 2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513095 2551 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:14.513098 2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.513104 2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.513229 2551 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.515454 2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.516319 2551 server.go:1019] "Starting client certificate rotation" Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.516419 2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 11:55:14.519344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.516505 2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 11:55:14.519704 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.519228 2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 11:55:14.522878 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.522853 2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 11:55:14.532186 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.532156 2551 log.go:25] "Validated CRI v1 runtime API" Sep 16 11:55:14.535655 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.535634 2551 log.go:25] "Validated CRI v1 image API" Sep 16 11:55:14.536550 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.536531 2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 11:55:14.539185 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.539160 2551 fs.go:135] Filesystem UUIDs: map[4a466c7f-47a3-49cf-84eb-85b1e72c09c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d5d925f2-0f94-41dc-a5df-1b41104614ee:/dev/nvme0n1p4] Sep 16 11:55:14.539279 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.539183 2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 11:55:14.544468 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.544346 2551 manager.go:216] Machine: {Timestamp:2026-09-16 11:55:14.54315899 +0000 UTC m=+0.328432548 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087572 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d4891ccb30f03e4a04d25d1234cb SystemUUID:ec28d489-1ccb-30f0-3e4a-04d25d1234cb BootID:236d288f-d558-4cc3-b292-8e820739fb3f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:52:5f:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:52:5f:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:81:9c:55:a6:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 11:55:14.544468 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.544461 2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 11:55:14.544660 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.544646 2551 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 11:55:14.547418 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.547378 2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 11:55:14.547601 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.547432 2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-32.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 11:55:14.548150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.548137 2551 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 11:55:14.548189 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.548154 2551 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 11:55:14.548189 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.548171 2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 11:55:14.548244 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.548187 2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 11:55:14.548989 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.548978 2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 11:55:14.550080 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.550063 2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 11:55:14.551783 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551771 2551 state_mem.go:36] "Initialized new in-memory state store" Sep 16 11:55:14.551914 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551905 2551 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 11:55:14.551948 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551932 2551 kubelet.go:486] "Attempting to sync node with API server" Sep 16 11:55:14.551948 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551942 2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 11:55:14.552005 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551957 2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 11:55:14.552005 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551971 2551 kubelet.go:398] "Adding apiserver pod source" Sep 16 11:55:14.552005 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.551983 2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 11:55:14.552402 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.552389 2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 11:55:14.552444 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.552410 2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 11:55:14.556034 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.556015 2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 11:55:14.557987 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.557969 2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 11:55:14.558058 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.558001 2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 11:55:14.559458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559445 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 11:55:14.559528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559464 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 11:55:14.559528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559473 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 11:55:14.559528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559481 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 11:55:14.559528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559507 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 11:55:14.559528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559515 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 11:55:14.559528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559530 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 11:55:14.559696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559536 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 11:55:14.559696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559544 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 11:55:14.559696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559557 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 11:55:14.559696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559570 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 11:55:14.559696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.559576 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 11:55:14.560308 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.560297 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 11:55:14.560337 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.560309 2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 11:55:14.563753 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.563728 2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-32.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 11:55:14.563854 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.563781 2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 11:55:14.563943 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.563910 2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 11:55:14.564525 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.564489 2551 server.go:1268] "Started kubelet" Sep 16 11:55:14.564586 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.564539 2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 11:55:14.565322 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.565252 2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 11:55:14.565452 ip-10-0-140-32 systemd[1]: Started Kubernetes Kubelet. Sep 16 11:55:14.568373 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.568342 2551 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 11:55:14.570075 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.570052 2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 11:55:14.570418 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.570394 2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 11:55:14.570527 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.570434 2551 server.go:310] "Adding debug handlers to kubelet server" Sep 16 11:55:14.574203 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.574138 2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 11:55:14.574748 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.574730 2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 11:55:14.574915 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.574897 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:14.575392 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.575374 2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 11:55:14.575392 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.575376 2551 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 11:55:14.575537 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.575405 2551 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 11:55:14.575537 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.574444 2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-32.ec2.internal.18d5cb1432f40e2c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-32.ec2.internal,UID:ip-10-0-140-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-32.ec2.internal,},FirstTimestamp:2026-09-16 11:55:14.564279852 +0000 UTC m=+0.349553411,LastTimestamp:2026-09-16 11:55:14.564279852 +0000 UTC m=+0.349553411,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-32.ec2.internal,}" Sep 16 11:55:14.575537 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.575523 2551 reconstruct.go:97] "Volume reconstruction finished" Sep 16 11:55:14.575537 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.575546 2551 reconciler.go:29] "Reconciler: start to sync state" Sep 16 11:55:14.578279 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578254 2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8mgzz" Sep 16 11:55:14.578471 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578456 2551 factory.go:55] Registering systemd factory Sep 16 11:55:14.578556 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578477 2551 factory.go:223] Registration of the systemd container factory successfully Sep 16 11:55:14.578556 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.578473 2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 11:55:14.578556 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.578455 2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-32.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 11:55:14.578750 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578730 2551 factory.go:153] Registering CRI-O factory Sep 16 11:55:14.578750 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578748 2551 factory.go:223] Registration of the crio container factory successfully Sep 16 11:55:14.578858 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578790 2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 11:55:14.578858 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578809 2551 factory.go:103] Registering Raw factory Sep 16 11:55:14.578858 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.578821 2551 manager.go:1196] Started watching for new ooms in manager Sep 16 11:55:14.579246 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.579235 2551 manager.go:349] Starting recovery of all containers Sep 16 11:55:14.584703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.584669 2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8mgzz" Sep 16 11:55:14.589766 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.589733 2551 manager.go:354] Recovery completed Sep 16 11:55:14.594873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.594851 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.597513 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.597482 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.597644 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.597533 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.597644 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.597549 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.598253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.598241 2551 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 11:55:14.598253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.598250 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.598325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.598254 2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 11:55:14.598325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.598275 2551 state_mem.go:36] "Initialized new in-memory state store" Sep 16 11:55:14.599549 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.599528 2551 policy_none.go:49] "None policy: Start" Sep 16 11:55:14.599648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.599573 2551 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 11:55:14.599648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.599590 2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 11:55:14.599920 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.599842 2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-32.ec2.internal.18d5cb1434ef29e5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-32.ec2.internal,UID:ip-10-0-140-32.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-32.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-32.ec2.internal,},FirstTimestamp:2026-09-16 11:55:14.597513701 +0000 UTC m=+0.382787267,LastTimestamp:2026-09-16 11:55:14.597513701 +0000 UTC m=+0.382787267,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-32.ec2.internal,}" Sep 16 11:55:14.600014 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.599962 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.600014 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.600001 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.600105 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.600024 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.601175 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.601150 2551 policy_none.go:47] "Start" Sep 16 11:55:14.635129 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635109 2551 manager.go:337] "Starting Device Plugin manager" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.635158 2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635171 2551 server.go:87] "Starting device plugin registration server" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635459 2551 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635470 2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635622 2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635714 2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.635724 2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.636197 2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 11:55:14.650999 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.636249 2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:14.708371 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.708316 2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 11:55:14.710067 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.710035 2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 11:55:14.710225 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.710140 2551 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 11:55:14.710225 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.710187 2551 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 11:55:14.710315 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.710253 2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 11:55:14.713099 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.713057 2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 11:55:14.736599 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.736547 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.737650 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.737629 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.737760 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.737666 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.737760 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.737678 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.737760 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.737711 2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.744357 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.744332 2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.744427 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.744367 2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-32.ec2.internal\": node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:14.770319 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.770207 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:14.811075 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.811035 2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal"] Sep 16 11:55:14.811166 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.811130 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.814673 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.814652 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.814782 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.814682 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.814782 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.814693 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.816153 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.816140 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.816315 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.816301 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.816393 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.816333 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.817038 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.817019 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.817161 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.817050 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.817161 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.817063 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.817161 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.817023 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.817161 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.817122 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.817161 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.817134 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.818380 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.818363 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.818466 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.818395 2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 11:55:14.819438 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.819251 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientMemory" Sep 16 11:55:14.819544 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.819461 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 11:55:14.819544 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.819477 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeHasSufficientPID" Sep 16 11:55:14.841690 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.841664 2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-32.ec2.internal\" not found" node="ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.846709 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.846690 2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-32.ec2.internal\" not found" node="ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.870454 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.870424 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:14.876756 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.876717 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37745f1482125bb6fede7a15de795bf8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal\" (UID: \"37745f1482125bb6fede7a15de795bf8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.876834 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.876762 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37745f1482125bb6fede7a15de795bf8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal\" (UID: \"37745f1482125bb6fede7a15de795bf8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.876834 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.876784 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956dcc432e8e3037c236189ed922ba03-config\") pod \"kube-apiserver-proxy-ip-10-0-140-32.ec2.internal\" (UID: \"956dcc432e8e3037c236189ed922ba03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.970758 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:14.970724 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:14.977017 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.976991 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37745f1482125bb6fede7a15de795bf8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal\" (UID: \"37745f1482125bb6fede7a15de795bf8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.977118 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.977022 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37745f1482125bb6fede7a15de795bf8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal\" (UID: \"37745f1482125bb6fede7a15de795bf8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.977118 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.977044 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956dcc432e8e3037c236189ed922ba03-config\") pod \"kube-apiserver-proxy-ip-10-0-140-32.ec2.internal\" (UID: \"956dcc432e8e3037c236189ed922ba03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.977118 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.977074 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37745f1482125bb6fede7a15de795bf8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal\" (UID: \"37745f1482125bb6fede7a15de795bf8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.977118 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.977093 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956dcc432e8e3037c236189ed922ba03-config\") pod \"kube-apiserver-proxy-ip-10-0-140-32.ec2.internal\" (UID: \"956dcc432e8e3037c236189ed922ba03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" Sep 16 11:55:14.977118 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:14.977094 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37745f1482125bb6fede7a15de795bf8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal\" (UID: \"37745f1482125bb6fede7a15de795bf8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:15.071545 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.071461 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.143971 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.143935 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:15.149655 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.149633 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" Sep 16 11:55:15.172422 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.171741 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.272288 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.272245 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.372906 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.372837 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.473402 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.473366 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.515824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.515801 2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 11:55:15.516223 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.515947 2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 11:55:15.573588 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.573559 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.589924 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.589854 2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:50:14 +0000 UTC" deadline="2028-05-07 18:55:56.722297226 +0000 UTC" Sep 16 11:55:15.589924 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.589909 2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14383h0m41.132392109s" Sep 16 11:55:15.620092 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.620046 2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 11:55:15.674036 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:15.673923 2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-32.ec2.internal\" not found" Sep 16 11:55:15.689576 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.689524 2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 11:55:15.774968 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.774943 2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" Sep 16 11:55:15.776157 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:15.776094 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37745f1482125bb6fede7a15de795bf8.slice/crio-fa6213b66b07e0d4103d535823eb5ecc9acf78d59b9e3545d0c34780f1bf2e44 WatchSource:0}: Error finding container fa6213b66b07e0d4103d535823eb5ecc9acf78d59b9e3545d0c34780f1bf2e44: Status 404 returned error can't find the container with id fa6213b66b07e0d4103d535823eb5ecc9acf78d59b9e3545d0c34780f1bf2e44 Sep 16 11:55:15.776392 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:15.776366 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956dcc432e8e3037c236189ed922ba03.slice/crio-f1f216da63d519b90a0c8ed447855a7d864a7e5e6d078379e2433155198e189d WatchSource:0}: Error finding container f1f216da63d519b90a0c8ed447855a7d864a7e5e6d078379e2433155198e189d: Status 404 returned error can't find the container with id f1f216da63d519b90a0c8ed447855a7d864a7e5e6d078379e2433155198e189d Sep 16 11:55:15.779525 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.779490 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 11:55:15.782925 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.782909 2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 11:55:15.784648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.784631 2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" Sep 16 11:55:15.792985 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:15.792960 2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 11:55:16.043954 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.043871 2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 11:55:16.392516 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.392472 2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 11:55:16.553234 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.553196 2551 apiserver.go:52] "Watching apiserver" Sep 16 11:55:16.561374 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.561342 2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 11:55:16.562492 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.562461 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-lvrwb","openshift-multus/multus-l2zhk","openshift-multus/network-metrics-daemon-76jr7","kube-system/konnectivity-agent-xrrn5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal","openshift-multus/multus-additional-cni-plugins-d2jbr","openshift-network-diagnostics/network-check-target-mvtdz","openshift-network-operator/iptables-alerter-gx9kz","openshift-ovn-kubernetes/ovnkube-node-wjpll","kube-system/global-pull-secret-syncer-62ntj","kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8","openshift-cluster-node-tuning-operator/tuned-kzlbn","openshift-dns/node-resolver-sdmwq"] Sep 16 11:55:16.565250 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.565194 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:16.565400 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.565299 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:16.567593 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.567562 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.569879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.569854 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:16.570023 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.569930 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:16.571308 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.571170 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.571308 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.571171 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 11:55:16.571308 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.571299 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.571580 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.571524 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f5mfp\"" Sep 16 11:55:16.572240 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.572215 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.575059 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.574388 2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 11:55:16.575059 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.574404 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.577171 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.576798 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.579618 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.579267 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.582092 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.581715 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.582725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.582702 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 11:55:16.582828 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.582717 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.582828 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.582791 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6kx8n\"" Sep 16 11:55:16.583026 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583008 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 11:55:16.583251 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583227 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.583582 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583558 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 11:55:16.583658 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583587 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.583715 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583678 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 11:55:16.584268 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583815 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-75zpq\"" Sep 16 11:55:16.584268 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.583979 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t4b7p\"" Sep 16 11:55:16.584268 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584088 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 11:55:16.584268 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584161 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 11:55:16.584268 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584223 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 11:55:16.584641 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584339 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.584641 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584349 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.584641 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584453 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 11:55:16.584641 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584582 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 11:55:16.584937 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584681 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 11:55:16.584937 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584737 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wdpwr\"" Sep 16 11:55:16.584937 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.584849 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.585281 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585259 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.585668 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585650 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d42e00ef-3df2-426e-8ab9-c55b73057716-host\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.585772 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585676 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr6nc\" (UniqueName: \"kubernetes.io/projected/d42e00ef-3df2-426e-8ab9-c55b73057716-kube-api-access-jr6nc\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.585772 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585741 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deb9a0a2-535a-4e0b-8dab-474b0cd2d70f-agent-certs\") pod \"konnectivity-agent-xrrn5\" (UID: \"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f\") " pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.585772 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585765 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-cnibin\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.585931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585781 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.585931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585805 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79c8g\" (UniqueName: \"kubernetes.io/projected/1962a963-727e-4a95-95fd-023b72e6eb48-kube-api-access-79c8g\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.585931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585829 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23cde59d-a1a1-4063-b447-4e5300a0f09e-host-slash\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.585931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585872 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deb9a0a2-535a-4e0b-8dab-474b0cd2d70f-konnectivity-ca\") pod \"konnectivity-agent-xrrn5\" (UID: \"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f\") " pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.585931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585902 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.585931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585913 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-system-cni-dir\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585958 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.585993 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586020 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23cde59d-a1a1-4063-b447-4e5300a0f09e-iptables-alerter-script\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586044 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgg56\" (UniqueName: \"kubernetes.io/projected/23cde59d-a1a1-4063-b447-4e5300a0f09e-kube-api-access-xgg56\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586077 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8tfr\" (UniqueName: \"kubernetes.io/projected/272abf77-006f-4a6d-933d-de30d31a1cf5-kube-api-access-b8tfr\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586112 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d42e00ef-3df2-426e-8ab9-c55b73057716-serviceca\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586144 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.586217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586175 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.586603 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.586239 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:16.588193 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.588167 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.589903 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.589800 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.590215 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.590187 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:16.592307 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.592283 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pj5lt\"" Sep 16 11:55:16.592529 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.592488 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pwpqg\"" Sep 16 11:55:16.592620 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.592550 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 11:55:16.593928 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.593905 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.595966 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.595942 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.600575 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.600546 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d96mf\"" Sep 16 11:55:16.602490 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.602464 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wdf9p\"" Sep 16 11:55:16.602712 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.602694 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.602848 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.602834 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.602908 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.602882 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 11:55:16.603047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.603033 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 11:55:16.603157 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.603034 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 11:55:16.607527 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.607400 2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 11:55:16.658718 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.658673 2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6q9dm" Sep 16 11:55:16.666527 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.666477 2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6q9dm" Sep 16 11:55:16.676405 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.676369 2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 11:55:16.686926 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.686892 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-conf-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.686939 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-system-cni-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.686963 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-etc-kubernetes\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.686986 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-systemd\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687008 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-sys\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687032 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1843d7ee-d758-4be5-9b19-b1de84c8ff62-tmp-dir\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687059 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-cnibin\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.687107 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687109 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-systemd-units\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687130 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-run-netns\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687151 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-host\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687173 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-system-cni-dir\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687178 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-cnibin\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687197 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687239 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23cde59d-a1a1-4063-b447-4e5300a0f09e-iptables-alerter-script\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687268 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-ovn\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687296 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687326 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-socket-dir-parent\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687352 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687373 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-node-log\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.687411 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687398 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d42e00ef-3df2-426e-8ab9-c55b73057716-host\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687420 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deb9a0a2-535a-4e0b-8dab-474b0cd2d70f-agent-certs\") pod \"konnectivity-agent-xrrn5\" (UID: \"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f\") " pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687443 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-cni-multus\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687475 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-daemon-config\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687527 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-system-cni-dir\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687559 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blv6n\" (UniqueName: \"kubernetes.io/projected/968602c3-6ae7-49f7-9c67-2770f01777d6-kube-api-access-blv6n\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687593 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysctl-d\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687606 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-run\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687622 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687645 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deb9a0a2-535a-4e0b-8dab-474b0cd2d70f-konnectivity-ca\") pod \"konnectivity-agent-xrrn5\" (UID: \"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f\") " pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687665 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-env-overrides\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687679 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-kubelet\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687697 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-var-lib-kubelet\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687719 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgg56\" (UniqueName: \"kubernetes.io/projected/23cde59d-a1a1-4063-b447-4e5300a0f09e-kube-api-access-xgg56\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687753 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-registration-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687769 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-device-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.687998 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687795 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px2xm\" (UniqueName: \"kubernetes.io/projected/38244eb8-3202-4a13-8e7e-aea8304450c0-kube-api-access-px2xm\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687820 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687838 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-cni-bin\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687855 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-hostroot\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687873 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da7c5f35-c4e1-419d-9c26-5e052ec5130c-kubelet-config\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687893 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-sys-fs\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687907 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovnkube-script-lib\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687920 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-k8s-cni-cncf-io\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687942 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da7c5f35-c4e1-419d-9c26-5e052ec5130c-dbus\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687960 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-modprobe-d\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.687999 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ba2517d-7a25-4d31-b469-d4f9c982d219-tmp\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688021 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt99h\" (UniqueName: \"kubernetes.io/projected/5ba2517d-7a25-4d31-b469-d4f9c982d219-kube-api-access-jt99h\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688043 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1843d7ee-d758-4be5-9b19-b1de84c8ff62-hosts-file\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688059 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688074 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-slash\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688087 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-log-socket\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.688680 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688092 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23cde59d-a1a1-4063-b447-4e5300a0f09e-iptables-alerter-script\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688104 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-cni-bin\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688122 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovn-node-metrics-cert\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688136 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-tuned\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688164 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23cde59d-a1a1-4063-b447-4e5300a0f09e-host-slash\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688261 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d42e00ef-3df2-426e-8ab9-c55b73057716-host\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688597 2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688649 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/deb9a0a2-535a-4e0b-8dab-474b0cd2d70f-konnectivity-ca\") pod \"konnectivity-agent-xrrn5\" (UID: \"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f\") " pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688763 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23cde59d-a1a1-4063-b447-4e5300a0f09e-host-slash\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688808 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-kubelet\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688849 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr44s\" (UniqueName: \"kubernetes.io/projected/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-kube-api-access-hr44s\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688876 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-cnibin\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688907 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-multus-certs\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688926 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1962a963-727e-4a95-95fd-023b72e6eb48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688934 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688961 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-cni-netd\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.688995 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-cni-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.689369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689033 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-netns\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689059 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysconfig\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689109 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-cni-binary-copy\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689121 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-kubernetes\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689146 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-lib-modules\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689173 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-socket-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689194 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysctl-conf\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689222 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m856q\" (UniqueName: \"kubernetes.io/projected/1843d7ee-d758-4be5-9b19-b1de84c8ff62-kube-api-access-m856q\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689252 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr6nc\" (UniqueName: \"kubernetes.io/projected/d42e00ef-3df2-426e-8ab9-c55b73057716-kube-api-access-jr6nc\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689302 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689374 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689477 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-var-lib-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689566 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-etc-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689615 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689648 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689683 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79c8g\" (UniqueName: \"kubernetes.io/projected/1962a963-727e-4a95-95fd-023b72e6eb48-kube-api-access-79c8g\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.690018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689720 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovnkube-config\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689825 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8tfr\" (UniqueName: \"kubernetes.io/projected/272abf77-006f-4a6d-933d-de30d31a1cf5-kube-api-access-b8tfr\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689856 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968602c3-6ae7-49f7-9c67-2770f01777d6-cni-binary-copy\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689894 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d42e00ef-3df2-426e-8ab9-c55b73057716-serviceca\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689925 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.689981 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.690009 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-systemd\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.690129 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.690211 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:17.190175788 +0000 UTC m=+2.975449355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.690329 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d42e00ef-3df2-426e-8ab9-c55b73057716-serviceca\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.690742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.690517 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1962a963-727e-4a95-95fd-023b72e6eb48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.692914 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.692885 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/deb9a0a2-535a-4e0b-8dab-474b0cd2d70f-agent-certs\") pod \"konnectivity-agent-xrrn5\" (UID: \"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f\") " pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.699886 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.699853 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:16.699886 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.699890 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:16.700105 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.699905 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:16.700105 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.699982 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:17.19996127 +0000 UTC m=+2.985234817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:16.710188 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.710150 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8tfr\" (UniqueName: \"kubernetes.io/projected/272abf77-006f-4a6d-933d-de30d31a1cf5-kube-api-access-b8tfr\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:16.710349 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.710233 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79c8g\" (UniqueName: \"kubernetes.io/projected/1962a963-727e-4a95-95fd-023b72e6eb48-kube-api-access-79c8g\") pod \"multus-additional-cni-plugins-d2jbr\" (UID: \"1962a963-727e-4a95-95fd-023b72e6eb48\") " pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.710779 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.710757 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr6nc\" (UniqueName: \"kubernetes.io/projected/d42e00ef-3df2-426e-8ab9-c55b73057716-kube-api-access-jr6nc\") pod \"node-ca-lvrwb\" (UID: \"d42e00ef-3df2-426e-8ab9-c55b73057716\") " pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.711340 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.711320 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgg56\" (UniqueName: \"kubernetes.io/projected/23cde59d-a1a1-4063-b447-4e5300a0f09e-kube-api-access-xgg56\") pod \"iptables-alerter-gx9kz\" (UID: \"23cde59d-a1a1-4063-b447-4e5300a0f09e\") " pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.716422 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.716364 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" event={"ID":"37745f1482125bb6fede7a15de795bf8","Type":"ContainerStarted","Data":"fa6213b66b07e0d4103d535823eb5ecc9acf78d59b9e3545d0c34780f1bf2e44"} Sep 16 11:55:16.717697 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.717672 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" event={"ID":"956dcc432e8e3037c236189ed922ba03","Type":"ContainerStarted","Data":"f1f216da63d519b90a0c8ed447855a7d864a7e5e6d078379e2433155198e189d"} Sep 16 11:55:16.790934 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.790896 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.790934 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.790943 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-socket-dir-parent\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791016 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791069 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-node-log\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791085 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-socket-dir-parent\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791112 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-cni-multus\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791142 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-daemon-config\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791147 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-node-log\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791165 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blv6n\" (UniqueName: \"kubernetes.io/projected/968602c3-6ae7-49f7-9c67-2770f01777d6-kube-api-access-blv6n\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791187 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysctl-d\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791185 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-cni-multus\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791216 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791207 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-run\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791257 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-run\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791295 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-env-overrides\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791327 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysctl-d\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791332 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-kubelet\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791363 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-kubelet\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791375 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-var-lib-kubelet\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791403 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-registration-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791424 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-device-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791446 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px2xm\" (UniqueName: \"kubernetes.io/projected/38244eb8-3202-4a13-8e7e-aea8304450c0-kube-api-access-px2xm\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791468 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791512 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-cni-bin\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791491 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-device-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791550 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-registration-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791554 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791561 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-var-lib-cni-bin\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791598 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-var-lib-kubelet\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.791696 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791603 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-hostroot\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791631 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da7c5f35-c4e1-419d-9c26-5e052ec5130c-kubelet-config\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791655 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-sys-fs\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791691 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovnkube-script-lib\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791713 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-k8s-cni-cncf-io\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791715 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-daemon-config\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791725 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-hostroot\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791737 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da7c5f35-c4e1-419d-9c26-5e052ec5130c-dbus\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791743 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da7c5f35-c4e1-419d-9c26-5e052ec5130c-kubelet-config\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791777 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-modprobe-d\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791794 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-env-overrides\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791802 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ba2517d-7a25-4d31-b469-d4f9c982d219-tmp\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791817 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-sys-fs\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791829 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt99h\" (UniqueName: \"kubernetes.io/projected/5ba2517d-7a25-4d31-b469-d4f9c982d219-kube-api-access-jt99h\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791847 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1843d7ee-d758-4be5-9b19-b1de84c8ff62-hosts-file\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791863 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da7c5f35-c4e1-419d-9c26-5e052ec5130c-dbus\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791867 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791898 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-slash\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.792432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791902 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-modprobe-d\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791779 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-k8s-cni-cncf-io\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791918 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-log-socket\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791923 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791940 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-cni-bin\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791953 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1843d7ee-d758-4be5-9b19-b1de84c8ff62-hosts-file\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791962 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovn-node-metrics-cert\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791977 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-log-socket\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791984 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-tuned\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.791997 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-slash\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792005 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-kubelet\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792018 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-cni-bin\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792028 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr44s\" (UniqueName: \"kubernetes.io/projected/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-kube-api-access-hr44s\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792050 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-cnibin\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792056 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-kubelet\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792071 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-multus-certs\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792096 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-cni-netd\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792118 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-cni-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.793280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792138 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-netns\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792160 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysconfig\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792171 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovnkube-script-lib\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792183 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-kubernetes\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792213 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-lib-modules\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792224 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-kubernetes\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792244 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-socket-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792262 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-netns\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792270 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysctl-conf\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792277 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-cni-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792297 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m856q\" (UniqueName: \"kubernetes.io/projected/1843d7ee-d758-4be5-9b19-b1de84c8ff62-kube-api-access-m856q\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792304 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysconfig\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792322 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792335 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-cni-netd\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792344 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-var-lib-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792361 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-lib-modules\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792365 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-etc-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794103 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792395 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792431 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792459 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovnkube-config\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792485 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968602c3-6ae7-49f7-9c67-2770f01777d6-cni-binary-copy\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792515 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-socket-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792508 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-host-run-multus-certs\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792397 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-etc-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792548 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/38244eb8-3202-4a13-8e7e-aea8304450c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792559 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-systemd\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792582 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-conf-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.792599 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792602 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-system-cni-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792621 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-etc-kubernetes\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:16.792649 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:55:17.292633536 +0000 UTC m=+3.077907093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792654 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-sysctl-conf\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792660 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-etc-kubernetes\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792665 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-systemd\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.794769 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792434 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-cnibin\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792708 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-multus-conf-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792740 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-var-lib-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792778 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-sys\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792807 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-systemd\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792841 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-systemd\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792886 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968602c3-6ae7-49f7-9c67-2770f01777d6-system-cni-dir\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792697 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-sys\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793066 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovnkube-config\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793083 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1843d7ee-d758-4be5-9b19-b1de84c8ff62-tmp-dir\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793111 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-systemd-units\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793126 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968602c3-6ae7-49f7-9c67-2770f01777d6-cni-binary-copy\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793137 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-run-netns\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793166 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-host\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793170 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-host-run-netns\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.792462 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-openvswitch\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793205 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-systemd-units\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793206 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-ovn\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793235 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-run-ovn\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793239 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5ba2517d-7a25-4d31-b469-d4f9c982d219-host\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.795844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.793433 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1843d7ee-d758-4be5-9b19-b1de84c8ff62-tmp-dir\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.795844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.794461 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ba2517d-7a25-4d31-b469-d4f9c982d219-tmp\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.795844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.794525 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-ovn-node-metrics-cert\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.795844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.794571 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5ba2517d-7a25-4d31-b469-d4f9c982d219-etc-tuned\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.801878 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.801853 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blv6n\" (UniqueName: \"kubernetes.io/projected/968602c3-6ae7-49f7-9c67-2770f01777d6-kube-api-access-blv6n\") pod \"multus-l2zhk\" (UID: \"968602c3-6ae7-49f7-9c67-2770f01777d6\") " pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.804871 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.804821 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px2xm\" (UniqueName: \"kubernetes.io/projected/38244eb8-3202-4a13-8e7e-aea8304450c0-kube-api-access-px2xm\") pod \"aws-ebs-csi-driver-node-wn2w8\" (UID: \"38244eb8-3202-4a13-8e7e-aea8304450c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:16.806533 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.806486 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m856q\" (UniqueName: \"kubernetes.io/projected/1843d7ee-d758-4be5-9b19-b1de84c8ff62-kube-api-access-m856q\") pod \"node-resolver-sdmwq\" (UID: \"1843d7ee-d758-4be5-9b19-b1de84c8ff62\") " pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.806754 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.806719 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt99h\" (UniqueName: \"kubernetes.io/projected/5ba2517d-7a25-4d31-b469-d4f9c982d219-kube-api-access-jt99h\") pod \"tuned-kzlbn\" (UID: \"5ba2517d-7a25-4d31-b469-d4f9c982d219\") " pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.806754 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.806731 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr44s\" (UniqueName: \"kubernetes.io/projected/0a8d0d18-4cc5-468d-9aea-a3612523ef6b-kube-api-access-hr44s\") pod \"ovnkube-node-wjpll\" (UID: \"0a8d0d18-4cc5-468d-9aea-a3612523ef6b\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.881050 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.881008 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gx9kz" Sep 16 11:55:16.891752 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.891723 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:16.904046 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.904016 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvrwb" Sep 16 11:55:16.911718 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.911695 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sdmwq" Sep 16 11:55:16.919425 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.919394 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" Sep 16 11:55:16.926190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.926163 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:16.932017 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.931992 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" Sep 16 11:55:16.941788 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.941755 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2zhk" Sep 16 11:55:16.947531 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:16.947504 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" Sep 16 11:55:17.196386 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.196302 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:17.196565 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.196462 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:17.196565 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.196548 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:18.196531265 +0000 UTC m=+3.981804813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:17.297360 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.297326 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:17.297553 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.297399 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:17.297553 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.297507 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:17.297553 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.297520 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:17.297553 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.297539 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:17.297553 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.297555 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:17.297747 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.297580 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:55:18.297560536 +0000 UTC m=+4.082834097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:17.297747 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.297599 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:18.297590375 +0000 UTC m=+4.082863923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:17.670000 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.669958 2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:50:16 +0000 UTC" deadline="2028-03-29 12:58:23.543392533 +0000 UTC" Sep 16 11:55:17.670000 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.669989 2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13441h3m5.873405927s" Sep 16 11:55:17.680201 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.680175 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8d0d18_4cc5_468d_9aea_a3612523ef6b.slice/crio-a8692d3ace1589f4cbcd5781397261cbb32c37708b4ece18edc1651dbee0d51a WatchSource:0}: Error finding container a8692d3ace1589f4cbcd5781397261cbb32c37708b4ece18edc1651dbee0d51a: Status 404 returned error can't find the container with id a8692d3ace1589f4cbcd5781397261cbb32c37708b4ece18edc1651dbee0d51a Sep 16 11:55:17.681741 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.681587 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb9a0a2_535a_4e0b_8dab_474b0cd2d70f.slice/crio-3a64c790de8bbc877b47252486b14509f7dc4c7976958fcba9e3788420de00b8 WatchSource:0}: Error finding container 3a64c790de8bbc877b47252486b14509f7dc4c7976958fcba9e3788420de00b8: Status 404 returned error can't find the container with id 3a64c790de8bbc877b47252486b14509f7dc4c7976958fcba9e3788420de00b8 Sep 16 11:55:17.684369 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.684349 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba2517d_7a25_4d31_b469_d4f9c982d219.slice/crio-c35a21955782d363709b49e81217cb29f3056fc62a565a06d8d8d3acb03603cf WatchSource:0}: Error finding container c35a21955782d363709b49e81217cb29f3056fc62a565a06d8d8d3acb03603cf: Status 404 returned error can't find the container with id c35a21955782d363709b49e81217cb29f3056fc62a565a06d8d8d3acb03603cf Sep 16 11:55:17.686004 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.685980 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968602c3_6ae7_49f7_9c67_2770f01777d6.slice/crio-46004a0d40b9be0218aa94b8dbd90fdb3bc6a15a5990557b4bd5e284678120f4 WatchSource:0}: Error finding container 46004a0d40b9be0218aa94b8dbd90fdb3bc6a15a5990557b4bd5e284678120f4: Status 404 returned error can't find the container with id 46004a0d40b9be0218aa94b8dbd90fdb3bc6a15a5990557b4bd5e284678120f4 Sep 16 11:55:17.686449 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.686430 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cde59d_a1a1_4063_b447_4e5300a0f09e.slice/crio-d0936eba89d7d5ea4dba431f3defad875d3c8aef0a23f4e790c821d0350e7507 WatchSource:0}: Error finding container d0936eba89d7d5ea4dba431f3defad875d3c8aef0a23f4e790c821d0350e7507: Status 404 returned error can't find the container with id d0936eba89d7d5ea4dba431f3defad875d3c8aef0a23f4e790c821d0350e7507 Sep 16 11:55:17.687773 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.687389 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42e00ef_3df2_426e_8ab9_c55b73057716.slice/crio-ab6c43d0fef551c2ab732da27c4ce3d2cba30e3e3c7ffd0d1a2b889abfb8fc2f WatchSource:0}: Error finding container ab6c43d0fef551c2ab732da27c4ce3d2cba30e3e3c7ffd0d1a2b889abfb8fc2f: Status 404 returned error can't find the container with id ab6c43d0fef551c2ab732da27c4ce3d2cba30e3e3c7ffd0d1a2b889abfb8fc2f Sep 16 11:55:17.688163 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.688069 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38244eb8_3202_4a13_8e7e_aea8304450c0.slice/crio-50b47166251cca1e637a6b02e474bec90f44e0f170784bef493779c34cc777d3 WatchSource:0}: Error finding container 50b47166251cca1e637a6b02e474bec90f44e0f170784bef493779c34cc777d3: Status 404 returned error can't find the container with id 50b47166251cca1e637a6b02e474bec90f44e0f170784bef493779c34cc777d3 Sep 16 11:55:17.690540 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.690515 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1843d7ee_d758_4be5_9b19_b1de84c8ff62.slice/crio-14dc70511ded5078259db0529b84db04f63ad8673680141b82ba687af809e80b WatchSource:0}: Error finding container 14dc70511ded5078259db0529b84db04f63ad8673680141b82ba687af809e80b: Status 404 returned error can't find the container with id 14dc70511ded5078259db0529b84db04f63ad8673680141b82ba687af809e80b Sep 16 11:55:17.690869 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:55:17.690772 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1962a963_727e_4a95_95fd_023b72e6eb48.slice/crio-bdb08243666dfad319c57d76d72ee9c09d73c3cbe5eb22d2e9724ef8a5e7ef37 WatchSource:0}: Error finding container bdb08243666dfad319c57d76d72ee9c09d73c3cbe5eb22d2e9724ef8a5e7ef37: Status 404 returned error can't find the container with id bdb08243666dfad319c57d76d72ee9c09d73c3cbe5eb22d2e9724ef8a5e7ef37 Sep 16 11:55:17.710445 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.710418 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:17.710591 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.710453 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:17.710591 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.710460 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:17.710687 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.710644 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:17.710782 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.710703 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:17.710782 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:17.710766 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:17.722133 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.722102 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerStarted","Data":"bdb08243666dfad319c57d76d72ee9c09d73c3cbe5eb22d2e9724ef8a5e7ef37"} Sep 16 11:55:17.725030 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.724993 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sdmwq" event={"ID":"1843d7ee-d758-4be5-9b19-b1de84c8ff62","Type":"ContainerStarted","Data":"14dc70511ded5078259db0529b84db04f63ad8673680141b82ba687af809e80b"} Sep 16 11:55:17.726959 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.726931 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvrwb" event={"ID":"d42e00ef-3df2-426e-8ab9-c55b73057716","Type":"ContainerStarted","Data":"ab6c43d0fef551c2ab732da27c4ce3d2cba30e3e3c7ffd0d1a2b889abfb8fc2f"} Sep 16 11:55:17.727978 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.727951 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gx9kz" event={"ID":"23cde59d-a1a1-4063-b447-4e5300a0f09e","Type":"ContainerStarted","Data":"d0936eba89d7d5ea4dba431f3defad875d3c8aef0a23f4e790c821d0350e7507"} Sep 16 11:55:17.729003 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.728983 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" event={"ID":"5ba2517d-7a25-4d31-b469-d4f9c982d219","Type":"ContainerStarted","Data":"c35a21955782d363709b49e81217cb29f3056fc62a565a06d8d8d3acb03603cf"} Sep 16 11:55:17.731109 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.731075 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" event={"ID":"38244eb8-3202-4a13-8e7e-aea8304450c0","Type":"ContainerStarted","Data":"50b47166251cca1e637a6b02e474bec90f44e0f170784bef493779c34cc777d3"} Sep 16 11:55:17.733684 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.733659 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2zhk" event={"ID":"968602c3-6ae7-49f7-9c67-2770f01777d6","Type":"ContainerStarted","Data":"46004a0d40b9be0218aa94b8dbd90fdb3bc6a15a5990557b4bd5e284678120f4"} Sep 16 11:55:17.735191 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.735173 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xrrn5" event={"ID":"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f","Type":"ContainerStarted","Data":"3a64c790de8bbc877b47252486b14509f7dc4c7976958fcba9e3788420de00b8"} Sep 16 11:55:17.736200 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:17.736174 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"a8692d3ace1589f4cbcd5781397261cbb32c37708b4ece18edc1651dbee0d51a"} Sep 16 11:55:18.206403 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.206106 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:18.206403 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.206370 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:18.206641 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.206441 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:20.206423439 +0000 UTC m=+5.991696983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:18.307728 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.307685 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:18.307938 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.307766 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:18.307938 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.307924 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:18.308061 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.307948 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:18.308061 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.307961 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:18.308061 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.308020 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:20.308001219 +0000 UTC m=+6.093274770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:18.308463 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.308441 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:18.308576 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:18.308518 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:55:20.308486495 +0000 UTC m=+6.093760058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:18.670354 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.670284 2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:50:16 +0000 UTC" deadline="2028-05-19 04:47:07.863146732 +0000 UTC" Sep 16 11:55:18.670354 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.670352 2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14656h51m49.192798723s" Sep 16 11:55:18.755458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.755420 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" event={"ID":"956dcc432e8e3037c236189ed922ba03","Type":"ContainerStarted","Data":"f84532941e987f3ace66434a7e6ec2fea5e70758fbf5ae8320b1dd836f7355d4"} Sep 16 11:55:18.770646 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.770607 2551 generic.go:356] "Generic (PLEG): container finished" podID="37745f1482125bb6fede7a15de795bf8" containerID="e3514405671d5b8fcd26801f45d4e32ffb72b71b3c544141c7872817b9aa939e" exitCode=0 Sep 16 11:55:18.770843 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.770677 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" event={"ID":"37745f1482125bb6fede7a15de795bf8","Type":"ContainerDied","Data":"e3514405671d5b8fcd26801f45d4e32ffb72b71b3c544141c7872817b9aa939e"} Sep 16 11:55:18.805243 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:18.804767 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-32.ec2.internal" podStartSLOduration=3.804731962 podStartE2EDuration="3.804731962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:18.776689012 +0000 UTC m=+4.561962581" watchObservedRunningTime="2026-09-16 11:55:18.804731962 +0000 UTC m=+4.590005535" Sep 16 11:55:19.711433 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:19.711395 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:19.711930 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:19.711608 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:19.712062 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:19.712040 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:19.712162 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:19.712142 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:19.712217 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:19.712198 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:19.712289 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:19.712271 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:19.807086 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:19.807042 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" event={"ID":"37745f1482125bb6fede7a15de795bf8","Type":"ContainerStarted","Data":"bb8bb2d6211c8d1ddf00b9fdb2dd354ee0e2ccd2455b1568846a1a302d55d314"} Sep 16 11:55:19.834039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:19.833965 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-32.ec2.internal" podStartSLOduration=4.833941849 podStartE2EDuration="4.833941849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:19.833704805 +0000 UTC m=+5.618978377" watchObservedRunningTime="2026-09-16 11:55:19.833941849 +0000 UTC m=+5.619215408" Sep 16 11:55:20.226544 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:20.226486 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:20.226777 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.226756 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:20.226848 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.226837 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:24.226815854 +0000 UTC m=+10.012089402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:20.327667 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:20.327628 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:20.327844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:20.327724 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:20.327926 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.327849 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:20.327926 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.327914 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:55:24.327895376 +0000 UTC m=+10.113168925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:20.328443 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.328421 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:20.328559 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.328451 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:20.328559 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.328469 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:20.328559 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:20.328532 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:24.32851642 +0000 UTC m=+10.113789972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:21.712057 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:21.710916 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:21.712057 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:21.711123 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:21.712057 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:21.711726 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:21.712057 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:21.711844 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:21.712057 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:21.711914 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:21.712057 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:21.711991 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:23.710926 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:23.710892 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:23.711393 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:23.711024 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:23.711393 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:23.711041 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:23.711393 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:23.711184 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:23.711393 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:23.711202 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:23.711393 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:23.711299 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:24.262623 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:24.262580 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:24.262876 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.262761 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:24.262876 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.262846 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:32.262825169 +0000 UTC m=+18.048098729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:24.363714 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:24.363656 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:24.363911 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:24.363749 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:24.363911 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.363829 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:24.363911 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.363871 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:24.363911 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.363889 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:24.363911 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.363902 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:24.363911 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.363910 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:55:32.363890242 +0000 UTC m=+18.149163802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:24.364207 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:24.363950 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:32.363934418 +0000 UTC m=+18.149207985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:25.711469 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:25.711425 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:25.711987 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:25.711633 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:25.711987 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:25.711760 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:25.711987 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:25.711899 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:25.711987 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:25.711938 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:25.712181 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:25.712032 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:27.711091 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:27.711056 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:27.711597 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:27.711057 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:27.711597 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:27.711211 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:27.711597 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:27.711061 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:27.711597 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:27.711266 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:27.711597 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:27.711356 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:29.711387 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:29.711341 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:29.711879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:29.711341 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:29.711879 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:29.711486 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:29.711879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:29.711520 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:29.711879 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:29.711603 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:29.711879 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:29.711673 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:31.711154 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:31.711116 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:31.711725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:31.711158 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:31.711725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:31.711158 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:31.711725 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:31.711257 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:31.711725 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:31.711338 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:31.711725 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:31.711432 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:32.328689 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:32.328655 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:32.328889 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.328778 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:32.328889 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.328849 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:48.328828165 +0000 UTC m=+34.114101723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:32.429132 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:32.429094 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:32.429335 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:32.429149 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:32.429335 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.429286 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:32.429444 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.429369 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:55:48.429350908 +0000 UTC m=+34.214624455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:32.429444 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.429297 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:32.429444 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.429405 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:32.429444 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.429429 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:32.429646 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:32.429476 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:55:48.429460943 +0000 UTC m=+34.214734495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:33.710806 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:33.710766 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:33.710806 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:33.710801 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:33.711314 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:33.710898 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:33.711314 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:33.710938 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:33.711314 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:33.711028 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:33.711314 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:33.711111 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:35.710905 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.710590 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:35.710905 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:35.710858 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:35.710905 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.710607 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:35.711454 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:35.710923 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:35.711454 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.710591 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:35.711454 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:35.710970 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:35.843954 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.843806 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" event={"ID":"5ba2517d-7a25-4d31-b469-d4f9c982d219","Type":"ContainerStarted","Data":"2849d5483da22a3a28f55e748799d9a79360fe0d74710bb8c43f71513b68f963"} Sep 16 11:55:35.846246 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.846194 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" event={"ID":"38244eb8-3202-4a13-8e7e-aea8304450c0","Type":"ContainerStarted","Data":"c67659b1b674e09aed397cbbe990676342df306910756f0d21fa6c09457971e5"} Sep 16 11:55:35.848578 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.848486 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2zhk" event={"ID":"968602c3-6ae7-49f7-9c67-2770f01777d6","Type":"ContainerStarted","Data":"e00e641508109995ccc8c47f8bdf2e79fc5dea69ce655633d7321642b4cfd571"} Sep 16 11:55:35.851179 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.851137 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xrrn5" event={"ID":"deb9a0a2-535a-4e0b-8dab-474b0cd2d70f","Type":"ContainerStarted","Data":"f235d9c6264822ba0ce6ae1001cce477b23de94e92466cdab1f030f18f9ec8aa"} Sep 16 11:55:35.853850 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.853801 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"175636068cfa33eb5270551865629c54da6291c6255b2860dd6bc4ca1221855d"} Sep 16 11:55:35.854051 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.853945 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"2051f8a0d0971f9b816813517d8b04de9d5847b10f8f06319efb34351bc1c43d"} Sep 16 11:55:35.855969 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.855925 2551 generic.go:356] "Generic (PLEG): container finished" podID="1962a963-727e-4a95-95fd-023b72e6eb48" containerID="4f89272a2ba266207b7be4259dc2cd4510da1db885f5650acc3964316b378ea3" exitCode=0 Sep 16 11:55:35.856181 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.856044 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerDied","Data":"4f89272a2ba266207b7be4259dc2cd4510da1db885f5650acc3964316b378ea3"} Sep 16 11:55:35.858818 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.858775 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sdmwq" event={"ID":"1843d7ee-d758-4be5-9b19-b1de84c8ff62","Type":"ContainerStarted","Data":"4f1faffe9f859d0abaecd742ba876a180fcf9741df4213820722bd8ffb02471a"} Sep 16 11:55:35.860979 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.860945 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvrwb" event={"ID":"d42e00ef-3df2-426e-8ab9-c55b73057716","Type":"ContainerStarted","Data":"385085c2a462256e3e89c7aa85cec9dca37ddb31448f3541e840b765463af90f"} Sep 16 11:55:35.870152 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.869962 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kzlbn" podStartSLOduration=4.066850303 podStartE2EDuration="21.869930532s" totalImagesPullingTime="17.803080229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:35.86903772 +0000 UTC m=+21.654311291" watchObservedRunningTime="2026-09-16 11:55:35.869930532 +0000 UTC m=+21.655204101" Sep 16 11:55:35.892001 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.891917 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xrrn5" podStartSLOduration=11.598562948 podStartE2EDuration="21.891885644s" totalImagesPullingTime="10.293322696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:35.891793632 +0000 UTC m=+21.677067202" watchObservedRunningTime="2026-09-16 11:55:35.891885644 +0000 UTC m=+21.677159216" Sep 16 11:55:35.909144 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.909058 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lvrwb" podStartSLOduration=4.11025095 podStartE2EDuration="21.90902993s" totalImagesPullingTime="17.79877898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:35.908235976 +0000 UTC m=+21.693509545" watchObservedRunningTime="2026-09-16 11:55:35.90902993 +0000 UTC m=+21.694303655" Sep 16 11:55:35.932282 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.932192 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l2zhk" podStartSLOduration=4.063102817 podStartE2EDuration="21.932155533s" totalImagesPullingTime="17.869052716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:35.931194629 +0000 UTC m=+21.716468199" watchObservedRunningTime="2026-09-16 11:55:35.932155533 +0000 UTC m=+21.717429102" Sep 16 11:55:35.959948 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:35.959851 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sdmwq" podStartSLOduration=4.163744225 podStartE2EDuration="21.959816562s" totalImagesPullingTime="17.796072337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:35.959576384 +0000 UTC m=+21.744849955" watchObservedRunningTime="2026-09-16 11:55:35.959816562 +0000 UTC m=+21.745090130" Sep 16 11:55:36.316848 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.316790 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:36.317853 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.317821 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:36.746665 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.746618 2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 11:55:36.865966 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.865839 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" event={"ID":"38244eb8-3202-4a13-8e7e-aea8304450c0","Type":"ContainerStarted","Data":"2b0f15b5b593addb51085aa10a7d1c08b9d4903fedaf3f3c46b0ec974e52a10f"} Sep 16 11:55:36.869887 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.869835 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"59363f667fb4c56a126b7e4536a6d8b1246cec8c51ffa618fca55bae229a2e43"} Sep 16 11:55:36.870081 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.869936 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"7d70eec4976a887a5c2800d999c4e32bd23cba5cd28f42fc977d24b7d51ddc3d"} Sep 16 11:55:36.870081 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.869994 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"c44e2260d6458e5675cc7d305833c44461c801f0b81eea3b7dde73c505747884"} Sep 16 11:55:36.870081 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.870040 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"e0bdf154022537811e11acd7b184ec5564a18a344b00a13e370d756dc0ac0517"} Sep 16 11:55:36.870550 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.870526 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:36.870812 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:36.870777 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xrrn5" Sep 16 11:55:37.649237 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.649086 2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T11:55:36.746643053Z","UUID":"2888948f-2535-4148-a490-4eed689cbea0","Handler":null,"Name":"","Endpoint":""} Sep 16 11:55:37.651282 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.651249 2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 11:55:37.651528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.651296 2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 11:55:37.710868 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.710831 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:37.710868 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.710856 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:37.711113 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:37.710967 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:37.711113 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:37.711072 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:37.711223 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.711155 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:37.711280 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:37.711249 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:37.874894 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.874842 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gx9kz" event={"ID":"23cde59d-a1a1-4063-b447-4e5300a0f09e","Type":"ContainerStarted","Data":"4154263bbaa2e64b28a20ba4d305beb5276db14efcce6738a447fcdecc1bab3c"} Sep 16 11:55:37.877723 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.877679 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" event={"ID":"38244eb8-3202-4a13-8e7e-aea8304450c0","Type":"ContainerStarted","Data":"a8e0c27704797cc026be166dd0f3e6eff0b4fa03db686e3c0022e26be9446171"} Sep 16 11:55:37.897382 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.897316 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gx9kz" podStartSLOduration=6.097216735 podStartE2EDuration="23.897295809s" totalImagesPullingTime="17.800079074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:37.89663232 +0000 UTC m=+23.681905889" watchObservedRunningTime="2026-09-16 11:55:37.897295809 +0000 UTC m=+23.682569377" Sep 16 11:55:37.921348 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:37.921237 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn2w8" podStartSLOduration=4.286134136 podStartE2EDuration="23.92120987s" totalImagesPullingTime="19.635075734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:37.920047061 +0000 UTC m=+23.705320629" watchObservedRunningTime="2026-09-16 11:55:37.92120987 +0000 UTC m=+23.706483440" Sep 16 11:55:38.887252 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:38.886295 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"a0cf74f92e687f128f8cc097f4cc608422eeb347dc4e984e9e695c1d584355fa"} Sep 16 11:55:39.710798 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:39.710751 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:39.711049 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:39.710766 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:39.711049 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:39.710890 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:39.711049 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:39.710765 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:39.711049 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:39.710986 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:39.711049 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:39.711033 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:40.892537 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.892333 2551 generic.go:356] "Generic (PLEG): container finished" podID="1962a963-727e-4a95-95fd-023b72e6eb48" containerID="82aaa9cf516a31cd73b85245417de8802b0c56364d0355a1e577b92108310704" exitCode=0 Sep 16 11:55:40.893432 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.892401 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerDied","Data":"82aaa9cf516a31cd73b85245417de8802b0c56364d0355a1e577b92108310704"} Sep 16 11:55:40.896987 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.896947 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" event={"ID":"0a8d0d18-4cc5-468d-9aea-a3612523ef6b","Type":"ContainerStarted","Data":"601811b67253f639cd004f9eea56b6356ad2491435c74d2345578fba6be2bc6a"} Sep 16 11:55:40.897249 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.897231 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:40.897310 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.897298 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:40.897366 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.897355 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:40.913844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.913820 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:40.913983 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.913925 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:55:40.952417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:40.952362 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" podStartSLOduration=9.078035379 podStartE2EDuration="26.952343401s" totalImagesPullingTime="17.874308022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:40.95212161 +0000 UTC m=+26.737395183" watchObservedRunningTime="2026-09-16 11:55:40.952343401 +0000 UTC m=+26.737617034" Sep 16 11:55:41.711200 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:41.711155 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:41.711200 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:41.711204 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:41.711468 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:41.711210 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:41.711468 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:41.711298 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:41.711468 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:41.711389 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:41.711468 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:41.711456 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:41.903081 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:41.902883 2551 generic.go:356] "Generic (PLEG): container finished" podID="1962a963-727e-4a95-95fd-023b72e6eb48" containerID="edba1d64b3deba52f1536664ff57284bda068a3a51f63cb9a19a78eb75684874" exitCode=0 Sep 16 11:55:41.903081 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:41.902920 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerDied","Data":"edba1d64b3deba52f1536664ff57284bda068a3a51f63cb9a19a78eb75684874"} Sep 16 11:55:42.366688 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.366647 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-76jr7"] Sep 16 11:55:42.366844 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.366820 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:42.366967 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:42.366945 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:42.367243 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.367218 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-62ntj"] Sep 16 11:55:42.367350 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.367317 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:42.367413 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:42.367392 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:42.379817 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.379734 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvtdz"] Sep 16 11:55:42.379970 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.379850 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:42.379970 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:42.379947 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:42.907959 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.907925 2551 generic.go:356] "Generic (PLEG): container finished" podID="1962a963-727e-4a95-95fd-023b72e6eb48" containerID="a2c0649505e0b89c68d6baedde978259510e2b245087e371f441a6833dcbfa05" exitCode=0 Sep 16 11:55:42.908482 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:42.908012 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerDied","Data":"a2c0649505e0b89c68d6baedde978259510e2b245087e371f441a6833dcbfa05"} Sep 16 11:55:43.711129 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:43.711088 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:43.711336 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:43.711223 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:44.714910 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:44.714833 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:44.715322 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:44.714951 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:44.715322 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:44.715040 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:44.715322 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:44.715162 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:44.794698 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:44.794674 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sdmwq_1843d7ee-d758-4be5-9b19-b1de84c8ff62/dns-node-resolver/0.log" Sep 16 11:55:45.711073 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:45.711034 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:45.711302 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:45.711177 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:45.751677 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:45.751545 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvrwb_d42e00ef-3df2-426e-8ab9-c55b73057716/node-ca/0.log" Sep 16 11:55:46.710962 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:46.710703 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:46.711148 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:46.710720 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:46.711148 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:46.711079 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:46.711269 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:46.711146 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:47.711258 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:47.711226 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:47.711736 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:47.711705 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:47.923392 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:47.923361 2551 generic.go:356] "Generic (PLEG): container finished" podID="1962a963-727e-4a95-95fd-023b72e6eb48" containerID="c8a1e7addcf624c46f9b62826cfdefa1e2aa53825f2e304a9ecffdf560900453" exitCode=0 Sep 16 11:55:47.923585 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:47.923413 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerDied","Data":"c8a1e7addcf624c46f9b62826cfdefa1e2aa53825f2e304a9ecffdf560900453"} Sep 16 11:55:48.360693 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.360574 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:48.360944 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.360744 2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:48.360944 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.360831 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs podName:272abf77-006f-4a6d-933d-de30d31a1cf5 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:20.360809839 +0000 UTC m=+66.146083385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs") pod "network-metrics-daemon-76jr7" (UID: "272abf77-006f-4a6d-933d-de30d31a1cf5") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 11:55:48.461190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.461147 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:48.461190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.461201 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:48.461401 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.461293 2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:48.461401 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.461367 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret podName:da7c5f35-c4e1-419d-9c26-5e052ec5130c nodeName:}" failed. No retries permitted until 2026-09-16 11:56:20.461349178 +0000 UTC m=+66.246622724 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret") pod "global-pull-secret-syncer-62ntj" (UID: "da7c5f35-c4e1-419d-9c26-5e052ec5130c") : object "kube-system"/"original-pull-secret" not registered Sep 16 11:55:48.461401 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.461301 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 11:55:48.461401 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.461403 2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 11:55:48.461565 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.461415 2551 projected.go:196] Error preparing data for projected volume kube-api-access-fpjnb for pod openshift-network-diagnostics/network-check-target-mvtdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:48.461565 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.461456 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb podName:8582a7de-79bf-418f-9925-6057c581cf49 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:20.46144442 +0000 UTC m=+66.246717980 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fpjnb" (UniqueName: "kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb") pod "network-check-target-mvtdz" (UID: "8582a7de-79bf-418f-9925-6057c581cf49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 11:55:48.710845 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.710805 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:48.711075 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.710806 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:48.711075 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.710923 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:48.711075 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:48.711012 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:48.928742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.928703 2551 generic.go:356] "Generic (PLEG): container finished" podID="1962a963-727e-4a95-95fd-023b72e6eb48" containerID="c0abeb78e3b54248b65db28bb6fb7eedbe739680bfabfd5e0c59323b667bf0c6" exitCode=0 Sep 16 11:55:48.929359 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:48.928755 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerDied","Data":"c0abeb78e3b54248b65db28bb6fb7eedbe739680bfabfd5e0c59323b667bf0c6"} Sep 16 11:55:49.710663 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:49.710633 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:49.710858 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:49.710752 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:49.934537 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:49.934492 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" event={"ID":"1962a963-727e-4a95-95fd-023b72e6eb48","Type":"ContainerStarted","Data":"23a6e590185d48eeb4eb8e9d8a2e1e3b98fc441dcb8d30c197ab0b3fd3a4ad86"} Sep 16 11:55:50.710958 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:50.710921 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:50.711160 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:50.710924 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:50.711160 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:50.711032 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:50.711160 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:50.711098 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:51.711211 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:51.711170 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:51.711689 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:51.711291 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:52.710841 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:52.710743 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:52.710841 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:52.710743 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:52.711080 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:52.710882 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:52.711080 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:52.710914 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:53.711288 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:53.711252 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:53.711719 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:53.711368 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:54.711415 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:54.711353 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:54.711415 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:54.711414 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:54.712049 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:54.711487 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:54.712049 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:54.711606 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:55.710403 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:55.710373 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:55.710627 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:55.710519 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:56.711381 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:56.711343 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:56.711848 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:56.711343 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:56.711848 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:56.711519 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:56.711848 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:56.711544 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:57.711121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:57.711062 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:57.711441 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:57.711260 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:55:58.711227 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:58.711182 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:55:58.711547 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:58.711181 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:55:58.711547 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:58.711348 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:55:58.711547 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:58.711371 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:55:59.711039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:55:59.710998 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:55:59.711223 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:55:59.711118 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:56:00.710991 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:00.710894 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:00.710991 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:00.710953 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:00.711383 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:00.711073 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:56:00.711383 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:00.711178 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:56:01.711042 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:01.711004 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:01.711461 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:01.711132 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:56:02.710693 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:02.710641 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:02.710974 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:02.710658 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:02.710974 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:02.710802 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:56:02.710974 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:02.710928 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:56:03.710673 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:03.710631 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:03.711057 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:03.710745 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:56:04.711384 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:04.711345 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:04.711919 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:04.711345 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:04.711919 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:04.711448 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:56:04.711919 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:04.711575 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:56:05.710514 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:05.710467 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:05.710736 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:05.710602 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:56:06.711269 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:06.711228 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:06.711771 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:06.711232 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:06.711771 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:06.711344 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62ntj" podUID="da7c5f35-c4e1-419d-9c26-5e052ec5130c" Sep 16 11:56:06.711771 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:06.711412 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mvtdz" podUID="8582a7de-79bf-418f-9925-6057c581cf49" Sep 16 11:56:07.711091 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:07.711046 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:07.711421 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:07.711188 2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-76jr7" podUID="272abf77-006f-4a6d-933d-de30d31a1cf5" Sep 16 11:56:08.424688 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.424661 2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-32.ec2.internal" event="NodeReady" Sep 16 11:56:08.424861 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.424791 2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 11:56:08.588655 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.588312 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d2jbr" podStartSLOduration=26.220054068 podStartE2EDuration="54.588291026s" totalImagesPullingTime="28.368236958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:55:49.962713531 +0000 UTC m=+35.747987098" watchObservedRunningTime="2026-09-16 11:56:08.588291026 +0000 UTC m=+54.373564595" Sep 16 11:56:08.589201 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.589179 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-krb84"] Sep 16 11:56:08.591369 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.591352 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.596090 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.596060 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kzcms"] Sep 16 11:56:08.597092 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.597072 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 11:56:08.597206 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.597163 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 11:56:08.597614 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.597591 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 11:56:08.597724 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.597631 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 11:56:08.598046 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.598027 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.601845 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.601822 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bhqb6\"" Sep 16 11:56:08.616212 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616178 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/288a1af8-0e73-4cd8-af08-6533e3ef390b-crio-socket\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.616373 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616219 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/288a1af8-0e73-4cd8-af08-6533e3ef390b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.616373 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616261 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/288a1af8-0e73-4cd8-af08-6533e3ef390b-data-volume\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.616373 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616304 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2833af4-8b90-4c8e-b710-03ca1f39c373-config-volume\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.616373 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616341 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rnd\" (UniqueName: \"kubernetes.io/projected/b2833af4-8b90-4c8e-b710-03ca1f39c373-kube-api-access-56rnd\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.616552 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616400 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnvnf\" (UniqueName: \"kubernetes.io/projected/288a1af8-0e73-4cd8-af08-6533e3ef390b-kube-api-access-nnvnf\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.616552 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616420 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2833af4-8b90-4c8e-b710-03ca1f39c373-tmp-dir\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.616552 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616454 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/288a1af8-0e73-4cd8-af08-6533e3ef390b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.616552 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.616480 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2833af4-8b90-4c8e-b710-03ca1f39c373-metrics-tls\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.618404 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.618372 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 11:56:08.618571 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.618456 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dsd6v\"" Sep 16 11:56:08.620737 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.620712 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krb84"] Sep 16 11:56:08.622745 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.622724 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 11:56:08.627770 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.627730 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kzcms"] Sep 16 11:56:08.672265 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.672230 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7dz4b"] Sep 16 11:56:08.674341 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.674320 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.677703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.677641 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 11:56:08.677703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.677641 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 11:56:08.687100 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.687080 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5qhxg\"" Sep 16 11:56:08.687291 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.687280 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 11:56:08.704254 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.704222 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dz4b"] Sep 16 11:56:08.710703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.710670 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:08.710873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.710675 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:08.713479 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.713459 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 11:56:08.713978 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.713956 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v2ltd\"" Sep 16 11:56:08.714108 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.714001 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 11:56:08.714181 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.714113 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 11:56:08.716713 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716681 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2833af4-8b90-4c8e-b710-03ca1f39c373-metrics-tls\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.716831 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716730 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7t5g\" (UniqueName: \"kubernetes.io/projected/94210cd9-b4b4-42fe-931d-6ddea66155ee-kube-api-access-w7t5g\") pod \"ingress-canary-7dz4b\" (UID: \"94210cd9-b4b4-42fe-931d-6ddea66155ee\") " pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.716831 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716772 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/288a1af8-0e73-4cd8-af08-6533e3ef390b-crio-socket\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.716831 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716798 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/288a1af8-0e73-4cd8-af08-6533e3ef390b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.716831 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716828 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/288a1af8-0e73-4cd8-af08-6533e3ef390b-data-volume\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.717047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716853 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2833af4-8b90-4c8e-b710-03ca1f39c373-config-volume\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.717047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716887 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94210cd9-b4b4-42fe-931d-6ddea66155ee-cert\") pod \"ingress-canary-7dz4b\" (UID: \"94210cd9-b4b4-42fe-931d-6ddea66155ee\") " pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.717047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716900 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/288a1af8-0e73-4cd8-af08-6533e3ef390b-crio-socket\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.717047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716921 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rnd\" (UniqueName: \"kubernetes.io/projected/b2833af4-8b90-4c8e-b710-03ca1f39c373-kube-api-access-56rnd\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.717047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.716998 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnvnf\" (UniqueName: \"kubernetes.io/projected/288a1af8-0e73-4cd8-af08-6533e3ef390b-kube-api-access-nnvnf\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.717047 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.717032 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2833af4-8b90-4c8e-b710-03ca1f39c373-tmp-dir\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.717330 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.717081 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/288a1af8-0e73-4cd8-af08-6533e3ef390b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.717535 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.717488 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2833af4-8b90-4c8e-b710-03ca1f39c373-tmp-dir\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.717668 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.717540 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/288a1af8-0e73-4cd8-af08-6533e3ef390b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.717668 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.717540 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b2833af4-8b90-4c8e-b710-03ca1f39c373-config-volume\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.717790 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.717752 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/288a1af8-0e73-4cd8-af08-6533e3ef390b-data-volume\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.721219 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.721195 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2833af4-8b90-4c8e-b710-03ca1f39c373-metrics-tls\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.721358 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.721232 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/288a1af8-0e73-4cd8-af08-6533e3ef390b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.754105 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.754071 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnvnf\" (UniqueName: \"kubernetes.io/projected/288a1af8-0e73-4cd8-af08-6533e3ef390b-kube-api-access-nnvnf\") pod \"insights-runtime-extractor-krb84\" (UID: \"288a1af8-0e73-4cd8-af08-6533e3ef390b\") " pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.754252 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.754081 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rnd\" (UniqueName: \"kubernetes.io/projected/b2833af4-8b90-4c8e-b710-03ca1f39c373-kube-api-access-56rnd\") pod \"dns-default-kzcms\" (UID: \"b2833af4-8b90-4c8e-b710-03ca1f39c373\") " pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.818001 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.817953 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7t5g\" (UniqueName: \"kubernetes.io/projected/94210cd9-b4b4-42fe-931d-6ddea66155ee-kube-api-access-w7t5g\") pod \"ingress-canary-7dz4b\" (UID: \"94210cd9-b4b4-42fe-931d-6ddea66155ee\") " pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.818190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.818009 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94210cd9-b4b4-42fe-931d-6ddea66155ee-cert\") pod \"ingress-canary-7dz4b\" (UID: \"94210cd9-b4b4-42fe-931d-6ddea66155ee\") " pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.820621 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.820589 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94210cd9-b4b4-42fe-931d-6ddea66155ee-cert\") pod \"ingress-canary-7dz4b\" (UID: \"94210cd9-b4b4-42fe-931d-6ddea66155ee\") " pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.832438 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.832408 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7t5g\" (UniqueName: \"kubernetes.io/projected/94210cd9-b4b4-42fe-931d-6ddea66155ee-kube-api-access-w7t5g\") pod \"ingress-canary-7dz4b\" (UID: \"94210cd9-b4b4-42fe-931d-6ddea66155ee\") " pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:08.901315 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.901278 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-krb84" Sep 16 11:56:08.907911 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.907880 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:08.983423 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:08.983135 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7dz4b" Sep 16 11:56:09.241672 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.241589 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-krb84"] Sep 16 11:56:09.253019 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:09.252975 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a1af8_0e73_4cd8_af08_6533e3ef390b.slice/crio-0ee42156cee1c177524c18ce28235eedc24d06b0c51d00184e2486627906fe16 WatchSource:0}: Error finding container 0ee42156cee1c177524c18ce28235eedc24d06b0c51d00184e2486627906fe16: Status 404 returned error can't find the container with id 0ee42156cee1c177524c18ce28235eedc24d06b0c51d00184e2486627906fe16 Sep 16 11:56:09.310985 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.310957 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7dz4b"] Sep 16 11:56:09.311468 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:09.311447 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94210cd9_b4b4_42fe_931d_6ddea66155ee.slice/crio-a894fec4253f0ea32650bbc1938f55f2916c2cdb91b04edd205a7e51de13b0a5 WatchSource:0}: Error finding container a894fec4253f0ea32650bbc1938f55f2916c2cdb91b04edd205a7e51de13b0a5: Status 404 returned error can't find the container with id a894fec4253f0ea32650bbc1938f55f2916c2cdb91b04edd205a7e51de13b0a5 Sep 16 11:56:09.336484 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.336448 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kzcms"] Sep 16 11:56:09.339699 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:09.339667 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2833af4_8b90_4c8e_b710_03ca1f39c373.slice/crio-55c4c548a25b9ad525c4c7db064896cf88ce207bb68ddea6c2e3f0c66fa6f089 WatchSource:0}: Error finding container 55c4c548a25b9ad525c4c7db064896cf88ce207bb68ddea6c2e3f0c66fa6f089: Status 404 returned error can't find the container with id 55c4c548a25b9ad525c4c7db064896cf88ce207bb68ddea6c2e3f0c66fa6f089 Sep 16 11:56:09.711397 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.711359 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:09.722072 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.721022 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gplpj\"" Sep 16 11:56:09.722072 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.721096 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 11:56:09.983985 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.983890 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kzcms" event={"ID":"b2833af4-8b90-4c8e-b710-03ca1f39c373","Type":"ContainerStarted","Data":"55c4c548a25b9ad525c4c7db064896cf88ce207bb68ddea6c2e3f0c66fa6f089"} Sep 16 11:56:09.985458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.985379 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dz4b" event={"ID":"94210cd9-b4b4-42fe-931d-6ddea66155ee","Type":"ContainerStarted","Data":"a894fec4253f0ea32650bbc1938f55f2916c2cdb91b04edd205a7e51de13b0a5"} Sep 16 11:56:09.987931 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.987899 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krb84" event={"ID":"288a1af8-0e73-4cd8-af08-6533e3ef390b","Type":"ContainerStarted","Data":"d234db67623f8aeb23bfeb47341dd725c6bcc64e938e8279a9465463f396fc55"} Sep 16 11:56:09.988072 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.987960 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krb84" event={"ID":"288a1af8-0e73-4cd8-af08-6533e3ef390b","Type":"ContainerStarted","Data":"9bd3bcd30075cc13a33361ca9844103783e01c293d565cf508889665e98193b4"} Sep 16 11:56:09.988072 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:09.987980 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krb84" event={"ID":"288a1af8-0e73-4cd8-af08-6533e3ef390b","Type":"ContainerStarted","Data":"0ee42156cee1c177524c18ce28235eedc24d06b0c51d00184e2486627906fe16"} Sep 16 11:56:10.034616 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.034577 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2"] Sep 16 11:56:10.036560 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.036526 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:10.038833 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.038692 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 11:56:10.038833 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.038708 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5954m\"" Sep 16 11:56:10.052556 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.051739 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2"] Sep 16 11:56:10.127970 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.127920 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb2e38f3-42ea-41e9-8a61-791f7e8d014d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nxwl2\" (UID: \"eb2e38f3-42ea-41e9-8a61-791f7e8d014d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:10.229258 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.229217 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb2e38f3-42ea-41e9-8a61-791f7e8d014d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nxwl2\" (UID: \"eb2e38f3-42ea-41e9-8a61-791f7e8d014d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:10.229452 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:10.229402 2551 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 16 11:56:10.229536 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:10.229482 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb2e38f3-42ea-41e9-8a61-791f7e8d014d-tls-certificates podName:eb2e38f3-42ea-41e9-8a61-791f7e8d014d nodeName:}" failed. No retries permitted until 2026-09-16 11:56:10.729462748 +0000 UTC m=+56.514736317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/eb2e38f3-42ea-41e9-8a61-791f7e8d014d-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-nxwl2" (UID: "eb2e38f3-42ea-41e9-8a61-791f7e8d014d") : secret "prometheus-operator-admission-webhook-tls" not found Sep 16 11:56:10.734343 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.734302 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb2e38f3-42ea-41e9-8a61-791f7e8d014d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nxwl2\" (UID: \"eb2e38f3-42ea-41e9-8a61-791f7e8d014d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:10.738073 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.738034 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/eb2e38f3-42ea-41e9-8a61-791f7e8d014d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-nxwl2\" (UID: \"eb2e38f3-42ea-41e9-8a61-791f7e8d014d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:10.948454 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:10.948408 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:12.838870 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:12.838756 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2"] Sep 16 11:56:12.843488 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:12.843455 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2e38f3_42ea_41e9_8a61_791f7e8d014d.slice/crio-294e81de8a89150fb29fe4dbb171a89f3003c10fde09531ba2ec40a97e6ddab2 WatchSource:0}: Error finding container 294e81de8a89150fb29fe4dbb171a89f3003c10fde09531ba2ec40a97e6ddab2: Status 404 returned error can't find the container with id 294e81de8a89150fb29fe4dbb171a89f3003c10fde09531ba2ec40a97e6ddab2 Sep 16 11:56:12.919571 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:12.919546 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjpll" Sep 16 11:56:12.997879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:12.997837 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7dz4b" event={"ID":"94210cd9-b4b4-42fe-931d-6ddea66155ee","Type":"ContainerStarted","Data":"1779bd25a54d3ee6f39eb80324fcb74f8477228bff32c5b70e4f4f4b3d84c9e2"} Sep 16 11:56:12.999023 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:12.999002 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" event={"ID":"eb2e38f3-42ea-41e9-8a61-791f7e8d014d","Type":"ContainerStarted","Data":"294e81de8a89150fb29fe4dbb171a89f3003c10fde09531ba2ec40a97e6ddab2"} Sep 16 11:56:13.001074 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.001046 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-krb84" event={"ID":"288a1af8-0e73-4cd8-af08-6533e3ef390b","Type":"ContainerStarted","Data":"d4cb6df7856016e5c656a0728476382bb70ad77ee9ef46c8d7eeee3e661a5394"} Sep 16 11:56:13.002812 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.002786 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kzcms" event={"ID":"b2833af4-8b90-4c8e-b710-03ca1f39c373","Type":"ContainerStarted","Data":"1e207ddbeb410a4265d0db4be839b14054453464fd5cc1530c71d717452c2808"} Sep 16 11:56:13.002928 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.002823 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kzcms" event={"ID":"b2833af4-8b90-4c8e-b710-03ca1f39c373","Type":"ContainerStarted","Data":"90758491fc94959c9ae745d07691469290a515291e82340a5028cfb898d93e50"} Sep 16 11:56:13.002972 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.002937 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:13.048662 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.048618 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7dz4b" podStartSLOduration=1.7884381029999998 podStartE2EDuration="5.048603046s" totalImagesPullingTime="3.260164943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:13.048389687 +0000 UTC m=+58.833663256" watchObservedRunningTime="2026-09-16 11:56:13.048603046 +0000 UTC m=+58.833876613" Sep 16 11:56:13.083634 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.083576 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kzcms" podStartSLOduration=1.866572718 podStartE2EDuration="5.083551198s" totalImagesPullingTime="3.21697848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:13.082664083 +0000 UTC m=+58.867937651" watchObservedRunningTime="2026-09-16 11:56:13.083551198 +0000 UTC m=+58.868824770" Sep 16 11:56:13.132568 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:13.132487 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-krb84" podStartSLOduration=1.9732013959999999 podStartE2EDuration="5.132461403s" totalImagesPullingTime="3.159260007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:56:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:13.131878891 +0000 UTC m=+58.917152459" watchObservedRunningTime="2026-09-16 11:56:13.132461403 +0000 UTC m=+58.917734971" Sep 16 11:56:14.009648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:14.009589 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" event={"ID":"eb2e38f3-42ea-41e9-8a61-791f7e8d014d","Type":"ContainerStarted","Data":"55f49cf2026709e672cd47c3d7d0a2248f241697c2e9ae4664aeb87d16224b26"} Sep 16 11:56:14.044211 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:14.044163 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" podStartSLOduration=2.979003436 podStartE2EDuration="4.044146424s" totalImagesPullingTime="1.065142988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:14.043471028 +0000 UTC m=+59.828744597" watchObservedRunningTime="2026-09-16 11:56:14.044146424 +0000 UTC m=+59.829419991" Sep 16 11:56:15.012743 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:15.012709 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:15.017409 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:15.017385 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-nxwl2" Sep 16 11:56:16.110786 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.110745 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp"] Sep 16 11:56:16.114749 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.114724 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.118094 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.118061 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 11:56:16.118738 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.118717 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 11:56:16.118825 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.118717 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 11:56:16.118825 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.118717 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 11:56:16.119010 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.118995 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 11:56:16.119065 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.119027 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ckhb7\"" Sep 16 11:56:16.136365 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.134669 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp"] Sep 16 11:56:16.175522 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.175451 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2jx\" (UniqueName: \"kubernetes.io/projected/5be80290-74ba-4c2e-9217-89f746883d65-kube-api-access-dx2jx\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.175703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.175575 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.175703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.175612 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.175703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.175659 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be80290-74ba-4c2e-9217-89f746883d65-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.276113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.276071 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.276208 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.276149 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be80290-74ba-4c2e-9217-89f746883d65-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.276208 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.276177 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2jx\" (UniqueName: \"kubernetes.io/projected/5be80290-74ba-4c2e-9217-89f746883d65-kube-api-access-dx2jx\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.276275 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.276233 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.276336 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:16.276323 2551 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 16 11:56:16.276390 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:16.276381 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-tls podName:5be80290-74ba-4c2e-9217-89f746883d65 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:16.776367516 +0000 UTC m=+62.561641082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-s6kmp" (UID: "5be80290-74ba-4c2e-9217-89f746883d65") : secret "prometheus-operator-tls" not found Sep 16 11:56:16.276945 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.276921 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5be80290-74ba-4c2e-9217-89f746883d65-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.278624 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.278599 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.287161 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.287128 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2jx\" (UniqueName: \"kubernetes.io/projected/5be80290-74ba-4c2e-9217-89f746883d65-kube-api-access-dx2jx\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.780330 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.780271 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:16.782795 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:16.782765 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5be80290-74ba-4c2e-9217-89f746883d65-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s6kmp\" (UID: \"5be80290-74ba-4c2e-9217-89f746883d65\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:17.024178 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:17.024141 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" Sep 16 11:56:17.295306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:17.295085 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp"] Sep 16 11:56:17.297854 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:17.297826 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be80290_74ba_4c2e_9217_89f746883d65.slice/crio-a61361f3d0e43a4af47ff6fda88077a44aa96feb1401d0418c3864e89ad3e254 WatchSource:0}: Error finding container a61361f3d0e43a4af47ff6fda88077a44aa96feb1401d0418c3864e89ad3e254: Status 404 returned error can't find the container with id a61361f3d0e43a4af47ff6fda88077a44aa96feb1401d0418c3864e89ad3e254 Sep 16 11:56:18.024569 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:18.024528 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" event={"ID":"5be80290-74ba-4c2e-9217-89f746883d65","Type":"ContainerStarted","Data":"a61361f3d0e43a4af47ff6fda88077a44aa96feb1401d0418c3864e89ad3e254"} Sep 16 11:56:20.033415 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.033376 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" event={"ID":"5be80290-74ba-4c2e-9217-89f746883d65","Type":"ContainerStarted","Data":"5289fd7cd92fb7578e8a63e95c2021ded246d55196546645718dbfba6ddc0cf2"} Sep 16 11:56:20.033829 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.033432 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" event={"ID":"5be80290-74ba-4c2e-9217-89f746883d65","Type":"ContainerStarted","Data":"e97b503bc727039ad46690559393c5da93523b68e82ac46835c8013e5a374f95"} Sep 16 11:56:20.049325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.049269 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s6kmp" podStartSLOduration=2.229558182 podStartE2EDuration="4.049253277s" totalImagesPullingTime="1.819695095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:20.048654752 +0000 UTC m=+65.833928320" watchObservedRunningTime="2026-09-16 11:56:20.049253277 +0000 UTC m=+65.834526845" Sep 16 11:56:20.410208 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.410168 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:20.412450 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.412430 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 11:56:20.423583 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.423546 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/272abf77-006f-4a6d-933d-de30d31a1cf5-metrics-certs\") pod \"network-metrics-daemon-76jr7\" (UID: \"272abf77-006f-4a6d-933d-de30d31a1cf5\") " pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:20.511032 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.510989 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:20.511193 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.511042 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:20.514488 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.514464 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 11:56:20.514565 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.514547 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 11:56:20.523865 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.523835 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da7c5f35-c4e1-419d-9c26-5e052ec5130c-original-pull-secret\") pod \"global-pull-secret-syncer-62ntj\" (UID: \"da7c5f35-c4e1-419d-9c26-5e052ec5130c\") " pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:20.523920 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.523879 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 11:56:20.526595 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.526577 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gplpj\"" Sep 16 11:56:20.534555 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.534529 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpjnb\" (UniqueName: \"kubernetes.io/projected/8582a7de-79bf-418f-9925-6057c581cf49-kube-api-access-fpjnb\") pod \"network-check-target-mvtdz\" (UID: \"8582a7de-79bf-418f-9925-6057c581cf49\") " pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:20.535241 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.535222 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-76jr7" Sep 16 11:56:20.722689 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.722602 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62ntj" Sep 16 11:56:20.734319 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.734296 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v2ltd\"" Sep 16 11:56:20.742300 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.742267 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:20.808819 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.808771 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-76jr7"] Sep 16 11:56:20.811932 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:20.811904 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272abf77_006f_4a6d_933d_de30d31a1cf5.slice/crio-fbcf19168cdf29f7d4b13c79ce4365605ecc92541455dabfde24d2875ff84961 WatchSource:0}: Error finding container fbcf19168cdf29f7d4b13c79ce4365605ecc92541455dabfde24d2875ff84961: Status 404 returned error can't find the container with id fbcf19168cdf29f7d4b13c79ce4365605ecc92541455dabfde24d2875ff84961 Sep 16 11:56:20.991175 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:20.991101 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-62ntj"] Sep 16 11:56:20.995915 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:20.995883 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7c5f35_c4e1_419d_9c26_5e052ec5130c.slice/crio-30d233f55a82b9ccce231ca295d753640b4f9e0b89f42e33019b403000dc87ee WatchSource:0}: Error finding container 30d233f55a82b9ccce231ca295d753640b4f9e0b89f42e33019b403000dc87ee: Status 404 returned error can't find the container with id 30d233f55a82b9ccce231ca295d753640b4f9e0b89f42e33019b403000dc87ee Sep 16 11:56:21.015726 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:21.015692 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mvtdz"] Sep 16 11:56:21.018793 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:21.018765 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8582a7de_79bf_418f_9925_6057c581cf49.slice/crio-8d987d64af003b4099e90c98aea8967c560a9426f8c0e2781cfe44ad6fef3c4e WatchSource:0}: Error finding container 8d987d64af003b4099e90c98aea8967c560a9426f8c0e2781cfe44ad6fef3c4e: Status 404 returned error can't find the container with id 8d987d64af003b4099e90c98aea8967c560a9426f8c0e2781cfe44ad6fef3c4e Sep 16 11:56:21.041387 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:21.041346 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-76jr7" event={"ID":"272abf77-006f-4a6d-933d-de30d31a1cf5","Type":"ContainerStarted","Data":"fbcf19168cdf29f7d4b13c79ce4365605ecc92541455dabfde24d2875ff84961"} Sep 16 11:56:21.042491 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:21.042462 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvtdz" event={"ID":"8582a7de-79bf-418f-9925-6057c581cf49","Type":"ContainerStarted","Data":"8d987d64af003b4099e90c98aea8967c560a9426f8c0e2781cfe44ad6fef3c4e"} Sep 16 11:56:21.043552 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:21.043524 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-62ntj" event={"ID":"da7c5f35-c4e1-419d-9c26-5e052ec5130c","Type":"ContainerStarted","Data":"30d233f55a82b9ccce231ca295d753640b4f9e0b89f42e33019b403000dc87ee"} Sep 16 11:56:22.049753 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.049628 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-76jr7" event={"ID":"272abf77-006f-4a6d-933d-de30d31a1cf5","Type":"ContainerStarted","Data":"73adc0a79f139301562f4d9286114dec208ee6a9450e1e4c9bd7bf8ec513c1f9"} Sep 16 11:56:22.049753 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.049699 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-76jr7" event={"ID":"272abf77-006f-4a6d-933d-de30d31a1cf5","Type":"ContainerStarted","Data":"8ee82f88d3486f84371f64b7d6212c32629d56f5f15b70920a466d61e7e9f2da"} Sep 16 11:56:22.066142 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.066081 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-76jr7" podStartSLOduration=67.166836618 podStartE2EDuration="1m8.066055203s" totalImagesPullingTime="899.218585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:22.064193299 +0000 UTC m=+67.849466884" watchObservedRunningTime="2026-09-16 11:56:22.066055203 +0000 UTC m=+67.851328773" Sep 16 11:56:22.476943 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.476565 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-77979"] Sep 16 11:56:22.479208 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.479175 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.482352 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.481688 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 11:56:22.482352 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.481747 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 11:56:22.482352 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.482142 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mwdb2\"" Sep 16 11:56:22.494087 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.493316 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm"] Sep 16 11:56:22.495853 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.495830 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-77979"] Sep 16 11:56:22.496007 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.495962 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.498336 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.498309 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 11:56:22.499525 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.499478 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 11:56:22.499957 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.499932 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 11:56:22.500101 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.500015 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-njb7b\"" Sep 16 11:56:22.510916 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.510328 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm"] Sep 16 11:56:22.519532 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.519483 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6xbmg"] Sep 16 11:56:22.521860 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.521834 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.523894 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.523871 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 11:56:22.524133 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.524117 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 11:56:22.524431 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.524407 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pk7d9\"" Sep 16 11:56:22.524719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.524696 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 11:56:22.526011 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.525990 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.526109 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526024 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lnx\" (UniqueName: \"kubernetes.io/projected/99f21fd7-3079-4913-827d-8924b3675bee-kube-api-access-d5lnx\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.526109 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526053 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/99f21fd7-3079-4913-827d-8924b3675bee-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.526109 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526078 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.526247 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526224 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.526303 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526264 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.526303 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526291 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbzh\" (UniqueName: \"kubernetes.io/projected/64c36bc7-5406-4ad4-b595-5cb154d913f9-kube-api-access-2dbzh\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.526404 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526339 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99f21fd7-3079-4913-827d-8924b3675bee-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.526404 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526364 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.526404 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.526385 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64c36bc7-5406-4ad4-b595-5cb154d913f9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.627612 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627571 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2fdbec5-569e-4c65-a0ae-d69c45a23132-metrics-client-ca\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.627612 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627624 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-textfile\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.627879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627653 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-tls\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.627879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627720 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lnx\" (UniqueName: \"kubernetes.io/projected/99f21fd7-3079-4913-827d-8924b3675bee-kube-api-access-d5lnx\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.627879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627762 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/99f21fd7-3079-4913-827d-8924b3675bee-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.627879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627787 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.627879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627821 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-root\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.627879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627853 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627903 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:22.627915 2551 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627933 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-sys\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.627961 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-wtmp\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:22.627980 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-tls podName:99f21fd7-3079-4913-827d-8924b3675bee nodeName:}" failed. No retries permitted until 2026-09-16 11:56:23.127959603 +0000 UTC m=+68.913233156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-nvzhm" (UID: "99f21fd7-3079-4913-827d-8924b3675bee") : secret "kube-state-metrics-tls" not found Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:22.628056 2551 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628089 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2pq\" (UniqueName: \"kubernetes.io/projected/d2fdbec5-569e-4c65-a0ae-d69c45a23132-kube-api-access-xj2pq\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:22.628104 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls podName:64c36bc7-5406-4ad4-b595-5cb154d913f9 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:23.128087426 +0000 UTC m=+68.913360972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-77979" (UID: "64c36bc7-5406-4ad4-b595-5cb154d913f9") : secret "openshift-state-metrics-tls" not found Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628134 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64c36bc7-5406-4ad4-b595-5cb154d913f9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.628174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628154 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.628648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628193 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.628648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628228 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99f21fd7-3079-4913-827d-8924b3675bee-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.628648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628265 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.628648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628282 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/99f21fd7-3079-4913-827d-8924b3675bee-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.628648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628295 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.628648 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628336 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbzh\" (UniqueName: \"kubernetes.io/projected/64c36bc7-5406-4ad4-b595-5cb154d913f9-kube-api-access-2dbzh\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.628940 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.628863 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64c36bc7-5406-4ad4-b595-5cb154d913f9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.629174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.629114 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.629730 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.629690 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99f21fd7-3079-4913-827d-8924b3675bee-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.631316 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.631257 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.632940 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.632909 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.642789 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.642763 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbzh\" (UniqueName: \"kubernetes.io/projected/64c36bc7-5406-4ad4-b595-5cb154d913f9-kube-api-access-2dbzh\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:22.644284 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.644244 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lnx\" (UniqueName: \"kubernetes.io/projected/99f21fd7-3079-4913-827d-8924b3675bee-kube-api-access-d5lnx\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:22.729839 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729736 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2fdbec5-569e-4c65-a0ae-d69c45a23132-metrics-client-ca\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.729839 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729791 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-textfile\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.729839 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729816 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-tls\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729862 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-root\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729913 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-sys\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729935 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-wtmp\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729965 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2pq\" (UniqueName: \"kubernetes.io/projected/d2fdbec5-569e-4c65-a0ae-d69c45a23132-kube-api-access-xj2pq\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:22.729973 2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.729973 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-root\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730005 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:22.730044 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-tls podName:d2fdbec5-569e-4c65-a0ae-d69c45a23132 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:23.23002461 +0000 UTC m=+69.015298160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-tls") pod "node-exporter-6xbmg" (UID: "d2fdbec5-569e-4c65-a0ae-d69c45a23132") : secret "node-exporter-tls" not found Sep 16 11:56:22.730113 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730090 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730625 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730130 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-wtmp\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730625 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730177 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2fdbec5-569e-4c65-a0ae-d69c45a23132-sys\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730625 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730275 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-textfile\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730625 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730389 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2fdbec5-569e-4c65-a0ae-d69c45a23132-metrics-client-ca\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.730625 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.730543 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-accelerators-collector-config\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.732669 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.732639 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:22.741856 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:22.741825 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2pq\" (UniqueName: \"kubernetes.io/projected/d2fdbec5-569e-4c65-a0ae-d69c45a23132-kube-api-access-xj2pq\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:23.012401 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.012318 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kzcms" Sep 16 11:56:23.134529 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.134465 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:23.135617 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.135059 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:23.135617 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:23.135170 2551 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 16 11:56:23.135617 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:23.135244 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls podName:64c36bc7-5406-4ad4-b595-5cb154d913f9 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:24.13522321 +0000 UTC m=+69.920496778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-77979" (UID: "64c36bc7-5406-4ad4-b595-5cb154d913f9") : secret "openshift-state-metrics-tls" not found Sep 16 11:56:23.137957 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.137931 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/99f21fd7-3079-4913-827d-8924b3675bee-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nvzhm\" (UID: \"99f21fd7-3079-4913-827d-8924b3675bee\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:23.236341 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.236297 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-tls\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:23.239289 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.239259 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d2fdbec5-569e-4c65-a0ae-d69c45a23132-node-exporter-tls\") pod \"node-exporter-6xbmg\" (UID: \"d2fdbec5-569e-4c65-a0ae-d69c45a23132\") " pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:23.407856 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.407808 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" Sep 16 11:56:23.433291 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.433248 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6xbmg" Sep 16 11:56:23.452086 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:23.452023 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fdbec5_569e_4c65_a0ae_d69c45a23132.slice/crio-7c596db2160443815a6102d428e1a4ce916c057cf31c4231daec1bfb26938bdf WatchSource:0}: Error finding container 7c596db2160443815a6102d428e1a4ce916c057cf31c4231daec1bfb26938bdf: Status 404 returned error can't find the container with id 7c596db2160443815a6102d428e1a4ce916c057cf31c4231daec1bfb26938bdf Sep 16 11:56:23.587854 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.587819 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:56:23.592293 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.590992 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.595679 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.594150 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 11:56:23.595679 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.594632 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 11:56:23.595679 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.594942 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k6nvs\"" Sep 16 11:56:23.595679 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.595078 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 11:56:23.595679 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.595230 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 11:56:23.596033 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.595985 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 11:56:23.596602 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.596359 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 11:56:23.596927 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.596910 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 11:56:23.596927 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.596921 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 11:56:23.597391 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.597372 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 11:56:23.612195 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.612121 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:56:23.639899 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.639849 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640074 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640005 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-out\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640074 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640041 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640219 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640071 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640219 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640098 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zwv\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-kube-api-access-h5zwv\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640219 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640144 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640219 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640170 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640219 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640209 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640252 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640411 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640454 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640488 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.640638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.640563 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-web-config\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.706693 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.706609 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm"] Sep 16 11:56:23.741999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.741960 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.741999 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742001 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.742242 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742029 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.742242 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742055 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742653 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-web-config\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:23.742667 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle podName:346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:24.24264205 +0000 UTC m=+70.027915616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5") : configmap references non-existent config key: ca-bundle.crt Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742721 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742780 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-out\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742815 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742848 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742874 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zwv\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-kube-api-access-h5zwv\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742936 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742951 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.742987 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.743923 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:23.744345 2551 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:23.744428 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls podName:346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:24.244407864 +0000 UTC m=+70.029681412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5") : secret "alertmanager-main-tls" not found Sep 16 11:56:23.744980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.744562 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.745859 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.745139 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.747260 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.747231 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.749278 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.749231 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.750150 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.750122 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-web-config\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.750631 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.750588 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-out\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.750747 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.750721 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.751154 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.751112 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-volume\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.751803 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.751632 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:23.759972 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:23.759911 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zwv\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-kube-api-access-h5zwv\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:24.060246 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.060155 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xbmg" event={"ID":"d2fdbec5-569e-4c65-a0ae-d69c45a23132","Type":"ContainerStarted","Data":"7c596db2160443815a6102d428e1a4ce916c057cf31c4231daec1bfb26938bdf"} Sep 16 11:56:24.146463 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.146422 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:24.149152 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.149125 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/64c36bc7-5406-4ad4-b595-5cb154d913f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-77979\" (UID: \"64c36bc7-5406-4ad4-b595-5cb154d913f9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:24.247122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.247079 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:24.247309 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.247180 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:24.247309 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:24.247288 2551 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 16 11:56:24.247439 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:24.247386 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls podName:346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5 nodeName:}" failed. No retries permitted until 2026-09-16 11:56:25.247364563 +0000 UTC m=+71.032638111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5") : secret "alertmanager-main-tls" not found Sep 16 11:56:24.248069 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.248035 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:24.292641 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:24.292601 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" Sep 16 11:56:25.256686 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.256643 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:25.259742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.259713 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:25.405609 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.405571 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:56:25.581700 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.581608 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-65774677d-bmjp9"] Sep 16 11:56:25.584528 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.584487 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.586683 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.586661 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2nk0cd9a2773r\"" Sep 16 11:56:25.586837 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.586661 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 11:56:25.586837 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.586666 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wrqhj\"" Sep 16 11:56:25.586837 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.586665 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 11:56:25.587010 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.586848 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 11:56:25.587257 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.587233 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 11:56:25.587360 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.587287 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 11:56:25.602584 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.602556 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65774677d-bmjp9"] Sep 16 11:56:25.661291 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661247 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-tls\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661322 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661358 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd7dfcfb-7551-483b-a213-c46a3a87aabc-metrics-client-ca\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661401 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661430 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-grpc-tls\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661465 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661453 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2gs\" (UniqueName: \"kubernetes.io/projected/cd7dfcfb-7551-483b-a213-c46a3a87aabc-kube-api-access-bw2gs\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661733 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661567 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.661733 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.661673 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762393 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762353 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762393 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762400 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-grpc-tls\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762716 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762419 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2gs\" (UniqueName: \"kubernetes.io/projected/cd7dfcfb-7551-483b-a213-c46a3a87aabc-kube-api-access-bw2gs\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762716 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762465 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762716 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762648 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762872 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762725 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-tls\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762872 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762792 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.762872 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.762828 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd7dfcfb-7551-483b-a213-c46a3a87aabc-metrics-client-ca\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.763671 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.763641 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd7dfcfb-7551-483b-a213-c46a3a87aabc-metrics-client-ca\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.765742 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.765690 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.765860 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.765794 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.765932 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.765872 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-grpc-tls\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.765982 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.765941 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.765982 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.765943 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.766344 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.766323 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cd7dfcfb-7551-483b-a213-c46a3a87aabc-secret-thanos-querier-tls\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.771311 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.771280 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2gs\" (UniqueName: \"kubernetes.io/projected/cd7dfcfb-7551-483b-a213-c46a3a87aabc-kube-api-access-bw2gs\") pod \"thanos-querier-65774677d-bmjp9\" (UID: \"cd7dfcfb-7551-483b-a213-c46a3a87aabc\") " pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.896039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:25.895955 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:25.981782 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:25.981744 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f21fd7_3079_4913_827d_8924b3675bee.slice/crio-e4afc89c140bb35fb08d29e49547b5d55d047d3fbbd63e1c4fb0d1c5c6a9cc9e WatchSource:0}: Error finding container e4afc89c140bb35fb08d29e49547b5d55d047d3fbbd63e1c4fb0d1c5c6a9cc9e: Status 404 returned error can't find the container with id e4afc89c140bb35fb08d29e49547b5d55d047d3fbbd63e1c4fb0d1c5c6a9cc9e Sep 16 11:56:26.071355 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.071307 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" event={"ID":"99f21fd7-3079-4913-827d-8924b3675bee","Type":"ContainerStarted","Data":"e4afc89c140bb35fb08d29e49547b5d55d047d3fbbd63e1c4fb0d1c5c6a9cc9e"} Sep 16 11:56:26.496175 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.496096 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-77979"] Sep 16 11:56:26.500047 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:26.500018 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c36bc7_5406_4ad4_b595_5cb154d913f9.slice/crio-e7f1920a247f30dc58cb24ad6719f2d7f8cd53194189f8c121956b98c6f6d642 WatchSource:0}: Error finding container e7f1920a247f30dc58cb24ad6719f2d7f8cd53194189f8c121956b98c6f6d642: Status 404 returned error can't find the container with id e7f1920a247f30dc58cb24ad6719f2d7f8cd53194189f8c121956b98c6f6d642 Sep 16 11:56:26.532068 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.531957 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-65774677d-bmjp9"] Sep 16 11:56:26.535304 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:26.535275 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7dfcfb_7551_483b_a213_c46a3a87aabc.slice/crio-900e3baf2e23c65076281f18417261a07a577481125bc9175230c8092189546b WatchSource:0}: Error finding container 900e3baf2e23c65076281f18417261a07a577481125bc9175230c8092189546b: Status 404 returned error can't find the container with id 900e3baf2e23c65076281f18417261a07a577481125bc9175230c8092189546b Sep 16 11:56:26.548155 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.548128 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:56:26.554825 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:26.554785 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346eaa8c_0224_4e16_b1aa_9dfa1f4e03c5.slice/crio-4f261712683d057dfe5da80d41b20ca28f716a7e97b220f92ed4671acdfb6341 WatchSource:0}: Error finding container 4f261712683d057dfe5da80d41b20ca28f716a7e97b220f92ed4671acdfb6341: Status 404 returned error can't find the container with id 4f261712683d057dfe5da80d41b20ca28f716a7e97b220f92ed4671acdfb6341 Sep 16 11:56:26.744760 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.744664 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-rpbrf"] Sep 16 11:56:26.747902 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.747841 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:26.751417 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.751391 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-q9brb\"" Sep 16 11:56:26.751589 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.751391 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 11:56:26.751715 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.751700 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 11:56:26.766140 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.766115 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rpbrf"] Sep 16 11:56:26.871850 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.871816 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9qq\" (UniqueName: \"kubernetes.io/projected/a3b72146-3cfe-499d-ac88-3e6ec0dec451-kube-api-access-gw9qq\") pod \"downloads-76f99584f4-rpbrf\" (UID: \"a3b72146-3cfe-499d-ac88-3e6ec0dec451\") " pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:26.972745 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.972714 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9qq\" (UniqueName: \"kubernetes.io/projected/a3b72146-3cfe-499d-ac88-3e6ec0dec451-kube-api-access-gw9qq\") pod \"downloads-76f99584f4-rpbrf\" (UID: \"a3b72146-3cfe-499d-ac88-3e6ec0dec451\") " pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:26.992125 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:26.992091 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9qq\" (UniqueName: \"kubernetes.io/projected/a3b72146-3cfe-499d-ac88-3e6ec0dec451-kube-api-access-gw9qq\") pod \"downloads-76f99584f4-rpbrf\" (UID: \"a3b72146-3cfe-499d-ac88-3e6ec0dec451\") " pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:27.025218 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.025134 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5bd97f8f7d-mvmds"] Sep 16 11:56:27.029635 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.029603 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.032771 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.032747 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 11:56:27.032919 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.032813 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 11:56:27.032988 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.032932 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4137it3vma1hl\"" Sep 16 11:56:27.033627 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.033608 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 11:56:27.033724 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.033642 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 11:56:27.037887 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.037867 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-28jd8\"" Sep 16 11:56:27.050703 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.050631 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bd97f8f7d-mvmds"] Sep 16 11:56:27.058995 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.058968 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:27.073486 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073453 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-secret-metrics-server-tls\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.073657 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073522 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e45b644-e98e-43ca-bf98-a370478cd7fa-metrics-server-audit-profiles\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.073657 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073559 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-secret-metrics-server-client-certs\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.073657 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073614 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qm2\" (UniqueName: \"kubernetes.io/projected/7e45b644-e98e-43ca-bf98-a370478cd7fa-kube-api-access-n7qm2\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.073826 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073700 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e45b644-e98e-43ca-bf98-a370478cd7fa-audit-log\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.073826 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073756 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e45b644-e98e-43ca-bf98-a370478cd7fa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.073826 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.073807 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-client-ca-bundle\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.080046 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.080010 2551 generic.go:356] "Generic (PLEG): container finished" podID="d2fdbec5-569e-4c65-a0ae-d69c45a23132" containerID="66fa2f147677c121443c23fa8d50c3ea7d9228f538d6585f42245d6b8bd48534" exitCode=0 Sep 16 11:56:27.080187 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.080113 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xbmg" event={"ID":"d2fdbec5-569e-4c65-a0ae-d69c45a23132","Type":"ContainerDied","Data":"66fa2f147677c121443c23fa8d50c3ea7d9228f538d6585f42245d6b8bd48534"} Sep 16 11:56:27.085929 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.085894 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" event={"ID":"64c36bc7-5406-4ad4-b595-5cb154d913f9","Type":"ContainerStarted","Data":"0eb466a78c80d0dcf1f7e074b68c2facf783ea275bf5a580f4ad6956ad409867"} Sep 16 11:56:27.086168 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.086141 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" event={"ID":"64c36bc7-5406-4ad4-b595-5cb154d913f9","Type":"ContainerStarted","Data":"23b11dbb3e2ae1a64d51903fd684c7ff5dce4477bb707b38b4ec56cf1e1a4416"} Sep 16 11:56:27.086289 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.086190 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" event={"ID":"64c36bc7-5406-4ad4-b595-5cb154d913f9","Type":"ContainerStarted","Data":"e7f1920a247f30dc58cb24ad6719f2d7f8cd53194189f8c121956b98c6f6d642"} Sep 16 11:56:27.088539 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.088479 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mvtdz" event={"ID":"8582a7de-79bf-418f-9925-6057c581cf49","Type":"ContainerStarted","Data":"d6fa6e7d07b4a748c59e52a6d3bb873630ca8c37c403b8544822ccd39f60db47"} Sep 16 11:56:27.088763 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.088736 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:56:27.090301 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.090271 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"900e3baf2e23c65076281f18417261a07a577481125bc9175230c8092189546b"} Sep 16 11:56:27.092078 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.092049 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-62ntj" event={"ID":"da7c5f35-c4e1-419d-9c26-5e052ec5130c","Type":"ContainerStarted","Data":"8ec4c87380dbfe485e9f076c24ca43a8532152a81c4fdfaeb8b75aef7d5dcc5b"} Sep 16 11:56:27.093403 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.093380 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"4f261712683d057dfe5da80d41b20ca28f716a7e97b220f92ed4671acdfb6341"} Sep 16 11:56:27.127845 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.127792 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mvtdz" podStartSLOduration=67.920376626 podStartE2EDuration="1m13.127769403s" totalImagesPullingTime="5.207392777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:27.126592868 +0000 UTC m=+72.911866433" watchObservedRunningTime="2026-09-16 11:56:27.127769403 +0000 UTC m=+72.913042972" Sep 16 11:56:27.146265 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.146209 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-62ntj" podStartSLOduration=67.919902681 podStartE2EDuration="1m13.146187497s" totalImagesPullingTime="5.226284816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 11:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:27.1445363 +0000 UTC m=+72.929809868" watchObservedRunningTime="2026-09-16 11:56:27.146187497 +0000 UTC m=+72.931461067" Sep 16 11:56:27.174397 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174363 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-secret-metrics-server-tls\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.174397 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174405 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e45b644-e98e-43ca-bf98-a370478cd7fa-metrics-server-audit-profiles\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.174655 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174445 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-secret-metrics-server-client-certs\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.174655 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174489 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qm2\" (UniqueName: \"kubernetes.io/projected/7e45b644-e98e-43ca-bf98-a370478cd7fa-kube-api-access-n7qm2\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.174765 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174685 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e45b644-e98e-43ca-bf98-a370478cd7fa-audit-log\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.174765 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174727 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e45b644-e98e-43ca-bf98-a370478cd7fa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.174864 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.174768 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-client-ca-bundle\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.175667 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.175299 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/7e45b644-e98e-43ca-bf98-a370478cd7fa-audit-log\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.175801 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.175771 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e45b644-e98e-43ca-bf98-a370478cd7fa-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.176199 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.176175 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/7e45b644-e98e-43ca-bf98-a370478cd7fa-metrics-server-audit-profiles\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.177299 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.177269 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-client-ca-bundle\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.177559 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.177537 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-secret-metrics-server-tls\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.177787 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.177768 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/7e45b644-e98e-43ca-bf98-a370478cd7fa-secret-metrics-server-client-certs\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.196332 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.196304 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qm2\" (UniqueName: \"kubernetes.io/projected/7e45b644-e98e-43ca-bf98-a370478cd7fa-kube-api-access-n7qm2\") pod \"metrics-server-5bd97f8f7d-mvmds\" (UID: \"7e45b644-e98e-43ca-bf98-a370478cd7fa\") " pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.274619 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.274589 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr"] Sep 16 11:56:27.279092 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.279031 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.281129 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.281106 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 11:56:27.281489 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.281474 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wgtdn\"" Sep 16 11:56:27.286618 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.286589 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr"] Sep 16 11:56:27.340548 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.340512 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:27.376980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.376780 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6r9b\" (UniqueName: \"kubernetes.io/projected/fbd818eb-3f3d-4ae1-999c-69d6817836fc-kube-api-access-d6r9b\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.376980 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.376894 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbd818eb-3f3d-4ae1-999c-69d6817836fc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.470901 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.470823 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rpbrf"] Sep 16 11:56:27.478072 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.477556 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbd818eb-3f3d-4ae1-999c-69d6817836fc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.478072 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.477641 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6r9b\" (UniqueName: \"kubernetes.io/projected/fbd818eb-3f3d-4ae1-999c-69d6817836fc-kube-api-access-d6r9b\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.478072 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:27.478029 2551 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 16 11:56:27.478323 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:56:27.478098 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbd818eb-3f3d-4ae1-999c-69d6817836fc-monitoring-plugin-cert podName:fbd818eb-3f3d-4ae1-999c-69d6817836fc nodeName:}" failed. No retries permitted until 2026-09-16 11:56:27.978076192 +0000 UTC m=+73.763349754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/fbd818eb-3f3d-4ae1-999c-69d6817836fc-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-46jrr" (UID: "fbd818eb-3f3d-4ae1-999c-69d6817836fc") : secret "monitoring-plugin-cert" not found Sep 16 11:56:27.485374 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:27.485341 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b72146_3cfe_499d_ac88_3e6ec0dec451.slice/crio-7fbd849dd98db78fc7770bd811ec4909522dee4fa865f04544c2cb9ed2001c1d WatchSource:0}: Error finding container 7fbd849dd98db78fc7770bd811ec4909522dee4fa865f04544c2cb9ed2001c1d: Status 404 returned error can't find the container with id 7fbd849dd98db78fc7770bd811ec4909522dee4fa865f04544c2cb9ed2001c1d Sep 16 11:56:27.491657 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.491545 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6r9b\" (UniqueName: \"kubernetes.io/projected/fbd818eb-3f3d-4ae1-999c-69d6817836fc-kube-api-access-d6r9b\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.646440 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.646357 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5bd97f8f7d-mvmds"] Sep 16 11:56:27.649393 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:27.649345 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e45b644_e98e_43ca_bf98_a370478cd7fa.slice/crio-63ecc6ab0a018ef7ca54ee5d391f14fa902504c981673e4a6dadf287f5b444b1 WatchSource:0}: Error finding container 63ecc6ab0a018ef7ca54ee5d391f14fa902504c981673e4a6dadf287f5b444b1: Status 404 returned error can't find the container with id 63ecc6ab0a018ef7ca54ee5d391f14fa902504c981673e4a6dadf287f5b444b1 Sep 16 11:56:27.982649 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.982511 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbd818eb-3f3d-4ae1-999c-69d6817836fc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:27.985507 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:27.985462 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fbd818eb-3f3d-4ae1-999c-69d6817836fc-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-46jrr\" (UID: \"fbd818eb-3f3d-4ae1-999c-69d6817836fc\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:28.101269 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.101229 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xbmg" event={"ID":"d2fdbec5-569e-4c65-a0ae-d69c45a23132","Type":"ContainerStarted","Data":"72ffc80a091bb8e790dee3527c76de2013409fc7bbf8696084511997e5766cb4"} Sep 16 11:56:28.101458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.101290 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6xbmg" event={"ID":"d2fdbec5-569e-4c65-a0ae-d69c45a23132","Type":"ContainerStarted","Data":"f0489f8b0a0b5e3d6b505fc55c7a7da2675c2be9d6abf514420d7bf847c754e8"} Sep 16 11:56:28.104410 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.104374 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" event={"ID":"99f21fd7-3079-4913-827d-8924b3675bee","Type":"ContainerStarted","Data":"efb2f0dd87d5898e01cf6aef8766fe2080f750167469997d07eb9ecc73c162ce"} Sep 16 11:56:28.104588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.104442 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" event={"ID":"99f21fd7-3079-4913-827d-8924b3675bee","Type":"ContainerStarted","Data":"13b7253a9901673b6cbad66068fbad8ce90ac8ade1d231f82ff867a8180feb0b"} Sep 16 11:56:28.104588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.104475 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" event={"ID":"99f21fd7-3079-4913-827d-8924b3675bee","Type":"ContainerStarted","Data":"f474d4ca92dcb3a0c71f424236ea2ee59a512207cef1e78634262eb79d366216"} Sep 16 11:56:28.106018 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.105987 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" event={"ID":"7e45b644-e98e-43ca-bf98-a370478cd7fa","Type":"ContainerStarted","Data":"63ecc6ab0a018ef7ca54ee5d391f14fa902504c981673e4a6dadf287f5b444b1"} Sep 16 11:56:28.107673 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.107631 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rpbrf" event={"ID":"a3b72146-3cfe-499d-ac88-3e6ec0dec451","Type":"ContainerStarted","Data":"7fbd849dd98db78fc7770bd811ec4909522dee4fa865f04544c2cb9ed2001c1d"} Sep 16 11:56:28.121684 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.121570 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6xbmg" podStartSLOduration=3.263546061 podStartE2EDuration="6.121547305s" totalImagesPullingTime="2.788418785s" totalInitContainerRuntime="69.582459ms" isStatefulPod=true podCreationTimestamp="2026-09-16 11:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:28.120363688 +0000 UTC m=+73.905637469" watchObservedRunningTime="2026-09-16 11:56:28.121547305 +0000 UTC m=+73.906820874" Sep 16 11:56:28.139784 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.139717 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nvzhm" podStartSLOduration=4.9179253 podStartE2EDuration="6.139692929s" totalImagesPullingTime="1.221767629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:28.138022104 +0000 UTC m=+73.923295673" watchObservedRunningTime="2026-09-16 11:56:28.139692929 +0000 UTC m=+73.924966498" Sep 16 11:56:28.206470 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.206429 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:28.657693 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.657611 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr"] Sep 16 11:56:28.784452 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.782444 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:56:28.787745 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.787718 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.798489 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.798727 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.798915 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.799116 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.799259 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.799389 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.799618 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.799863 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 11:56:28.800922 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.800787 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 11:56:28.801797 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.801753 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 11:56:28.802166 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.801914 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 11:56:28.802166 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.802035 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 11:56:28.802166 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.802071 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9pi4ipnvlo06g\"" Sep 16 11:56:28.802166 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.802097 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 11:56:28.802472 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.802226 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sjlg7\"" Sep 16 11:56:28.806843 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.806822 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 11:56:28.815306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.812669 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891454 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891534 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891564 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891598 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891636 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891685 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891714 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891743 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891767 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891793 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891858 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891892 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.891963 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.891969 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config-out\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892017 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892041 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892063 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-web-config\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892102 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grbmb\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-kube-api-access-grbmb\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892126 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892151 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.892824 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.892176 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993623 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993678 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993708 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993735 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993763 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993786 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993810 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993843 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config-out\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993888 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993911 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993934 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-web-config\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.993974 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grbmb\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-kube-api-access-grbmb\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994003 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994026 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994053 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994097 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.995039 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994130 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.996725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994154 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.996725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994182 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.996725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994224 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.996725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.994994 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.996725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.996142 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.996725 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.996365 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.997088 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.997060 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.998048 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.998021 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:28.999852 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:28.999660 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.001773 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.001201 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.001773 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.001360 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.001773 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.001692 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.004178 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.004136 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config-out\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.004556 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.004511 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-web-config\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.006683 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.006637 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.007281 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.007260 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grbmb\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-kube-api-access-grbmb\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.007402 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.007377 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.008232 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.008175 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.008232 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.008197 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.009472 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.008889 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.009472 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.008993 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.009472 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.009429 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.010463 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.010442 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.106544 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:29.106246 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:29.281559 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:29.281517 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd818eb_3f3d_4ae1_999c_69d6817836fc.slice/crio-fac00aa3e6d2178300942b224639cc0a994e20781ae71cdb76f6007907fa66c9 WatchSource:0}: Error finding container fac00aa3e6d2178300942b224639cc0a994e20781ae71cdb76f6007907fa66c9: Status 404 returned error can't find the container with id fac00aa3e6d2178300942b224639cc0a994e20781ae71cdb76f6007907fa66c9 Sep 16 11:56:30.083281 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.083230 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:56:30.087118 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:56:30.086876 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94d5738_7ffd_465d_9ce7_9ba28dfc5caf.slice/crio-03d21c19a092ae8aafb47b1796302192c0f3466b9703e997f1c530eea726cd89 WatchSource:0}: Error finding container 03d21c19a092ae8aafb47b1796302192c0f3466b9703e997f1c530eea726cd89: Status 404 returned error can't find the container with id 03d21c19a092ae8aafb47b1796302192c0f3466b9703e997f1c530eea726cd89 Sep 16 11:56:30.128461 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.128422 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" event={"ID":"64c36bc7-5406-4ad4-b595-5cb154d913f9","Type":"ContainerStarted","Data":"8f95b393206b74382b6621a2d571c70ab818c7d94942e89eb95569ded86d07f3"} Sep 16 11:56:30.131892 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.130943 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"03d21c19a092ae8aafb47b1796302192c0f3466b9703e997f1c530eea726cd89"} Sep 16 11:56:30.132873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.132738 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" event={"ID":"fbd818eb-3f3d-4ae1-999c-69d6817836fc","Type":"ContainerStarted","Data":"fac00aa3e6d2178300942b224639cc0a994e20781ae71cdb76f6007907fa66c9"} Sep 16 11:56:30.136338 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.136309 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"2c024730f14bb411c9c9cf3b6ade8947916c28774df2b8376f472004bb3cb4f1"} Sep 16 11:56:30.136457 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.136387 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"24cf549de38b7f83a92e709fd2c2ee8a4d5b8b62d658e5468d025c6b8ddbb5ed"} Sep 16 11:56:30.136457 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.136427 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"4d8b4c9ea3a76b3c746f4c45a826bf9ec2768927e51bd1b3a89b028499f9ffaa"} Sep 16 11:56:30.139132 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.139084 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e" exitCode=0 Sep 16 11:56:30.139237 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.139176 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e"} Sep 16 11:56:30.142078 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.142040 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" event={"ID":"7e45b644-e98e-43ca-bf98-a370478cd7fa","Type":"ContainerStarted","Data":"be645028d71ff8fcb57eb86fa519f5cf3089e2ffc72a117f868d8ec2a4cfee3d"} Sep 16 11:56:30.148167 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.148074 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-77979" podStartSLOduration=6.415637283 podStartE2EDuration="8.148052735s" totalImagesPullingTime="1.732415452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:30.14565918 +0000 UTC m=+75.930932760" watchObservedRunningTime="2026-09-16 11:56:30.148052735 +0000 UTC m=+75.933326307" Sep 16 11:56:30.189078 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:30.189012 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" podStartSLOduration=2.044297562 podStartE2EDuration="4.188987021s" totalImagesPullingTime="2.144689459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:30.187358842 +0000 UTC m=+75.972632411" watchObservedRunningTime="2026-09-16 11:56:30.188987021 +0000 UTC m=+75.974260590" Sep 16 11:56:31.151027 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:31.150853 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="b13b52905e58367f4c558f4081f19f08d0100b8430b9a49e977eaec5c79f8897" exitCode=0 Sep 16 11:56:31.152178 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:31.152154 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"b13b52905e58367f4c558f4081f19f08d0100b8430b9a49e977eaec5c79f8897"} Sep 16 11:56:32.163250 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.163210 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"f7673bcf2ced09991582d1eef9d2b9f362c9f211444ae9f8a57a57731b2b16c0"} Sep 16 11:56:32.163773 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.163316 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"2b29da090f74b9e7219df7321daa6ec157ced4a0999f8a12147353e306cb5507"} Sep 16 11:56:32.163773 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.163362 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" event={"ID":"cd7dfcfb-7551-483b-a213-c46a3a87aabc","Type":"ContainerStarted","Data":"83aa7f2f515d60a6df3f39fcfd4146009ab9fe3f5fb82c9c2c0f53d84ccc026c"} Sep 16 11:56:32.163773 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.163695 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:32.166263 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.166232 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" event={"ID":"fbd818eb-3f3d-4ae1-999c-69d6817836fc","Type":"ContainerStarted","Data":"b80c6c1dec74dc1425342b55f9bf301ac9925ac7fc817319c3e2733903816620"} Sep 16 11:56:32.166740 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.166708 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:32.172974 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.172946 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" Sep 16 11:56:32.186884 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.186811 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" podStartSLOduration=2.809180181 podStartE2EDuration="7.186790431s" totalImagesPullingTime="4.37761025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:32.186348741 +0000 UTC m=+77.971622335" watchObservedRunningTime="2026-09-16 11:56:32.186790431 +0000 UTC m=+77.972064000" Sep 16 11:56:32.207404 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:32.207325 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-46jrr" podStartSLOduration=3.502784288 podStartE2EDuration="5.207307403s" totalImagesPullingTime="1.704523115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:32.205437895 +0000 UTC m=+77.990711463" watchObservedRunningTime="2026-09-16 11:56:32.207307403 +0000 UTC m=+77.992580972" Sep 16 11:56:33.175845 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.175804 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884"} Sep 16 11:56:33.176286 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.175892 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161"} Sep 16 11:56:33.176286 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.175941 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19"} Sep 16 11:56:33.176286 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.175980 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534"} Sep 16 11:56:33.176286 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.176018 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e"} Sep 16 11:56:33.176286 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.176045 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerStarted","Data":"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66"} Sep 16 11:56:33.213928 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:33.209968 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=6.251405279 podStartE2EDuration="10.20994512s" totalImagesPullingTime="3.931281204s" totalInitContainerRuntime="27.258637ms" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:33.206728153 +0000 UTC m=+78.992001722" watchObservedRunningTime="2026-09-16 11:56:33.20994512 +0000 UTC m=+78.995218689" Sep 16 11:56:36.200834 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.200792 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"c528e384e03f83ad466895009b2ea02c446421a383f4e0dff6af3cc48b0c5617"} Sep 16 11:56:36.201253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.200912 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"e29421072278fe3d8f495eb8f12fb078bedc0ee2c73cde3db5edbc2ae49af54f"} Sep 16 11:56:36.201253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.200955 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"02ebbc7f9f72f78005532bd6749ec6282cffc3f6aa3b29872a56a045ad7c08c8"} Sep 16 11:56:36.201253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.200981 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"902b65c8eee0a19e2b9b48ee49b32794bc084a396fc829820cb01e1f1fac5401"} Sep 16 11:56:36.201253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.201023 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"d00c8b65ada8afebb195f105bd47f761bce2c782ff534b52938279b58e191f6c"} Sep 16 11:56:36.201253 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.201071 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerStarted","Data":"1a1a2c679fffca12228e5d94b5446dbc21e4c3b08d7b2aa536d5b62251d3c7f7"} Sep 16 11:56:36.228849 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:36.228774 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.201341184 podStartE2EDuration="8.228751516s" totalImagesPullingTime="4.005022396s" totalInitContainerRuntime="22.387936ms" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:36.228031713 +0000 UTC m=+82.013305281" watchObservedRunningTime="2026-09-16 11:56:36.228751516 +0000 UTC m=+82.014025085" Sep 16 11:56:38.183723 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:38.183692 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-65774677d-bmjp9" Sep 16 11:56:39.107339 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:39.107296 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:56:47.259749 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:47.259671 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rpbrf" event={"ID":"a3b72146-3cfe-499d-ac88-3e6ec0dec451","Type":"ContainerStarted","Data":"70965a6eeaeeb0ae1f581750ed31ef8e85f4547108bf4d8d9e348ab734ab711b"} Sep 16 11:56:47.260426 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:47.260404 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:47.279832 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:47.279574 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-rpbrf" Sep 16 11:56:47.281956 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:47.281894 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-rpbrf" podStartSLOduration=2.621383873 podStartE2EDuration="21.281873827s" totalImagesPullingTime="18.660489954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 11:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:56:47.277902173 +0000 UTC m=+93.063175770" watchObservedRunningTime="2026-09-16 11:56:47.281873827 +0000 UTC m=+93.067147395" Sep 16 11:56:47.340896 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:47.340861 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:47.341169 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:47.341151 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:56:48.302025 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:48.301990 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dz4b_94210cd9-b4b4-42fe-931d-6ddea66155ee/serve-healthcheck-canary/0.log" Sep 16 11:56:58.111227 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:56:58.111175 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mvtdz" Sep 16 11:57:07.347245 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:07.347212 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:57:07.351147 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:07.351117 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5bd97f8f7d-mvmds" Sep 16 11:57:29.107190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:29.107146 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:29.166353 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:29.166326 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:29.472716 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:29.472657 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:42.736819 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.736784 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:57:42.737340 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.737175 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="alertmanager" containerID="cri-o://7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66" gracePeriod=120 Sep 16 11:57:42.737340 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.737231 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-web" containerID="cri-o://e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534" gracePeriod=120 Sep 16 11:57:42.737340 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.737257 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="config-reloader" containerID="cri-o://b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e" gracePeriod=120 Sep 16 11:57:42.737340 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.737278 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy" containerID="cri-o://57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19" gracePeriod=120 Sep 16 11:57:42.737340 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.737317 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="prom-label-proxy" containerID="cri-o://b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884" gracePeriod=120 Sep 16 11:57:42.737641 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:42.737229 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-metric" containerID="cri-o://ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161" gracePeriod=120 Sep 16 11:57:43.514259 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514225 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884" exitCode=0 Sep 16 11:57:43.514259 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514252 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19" exitCode=0 Sep 16 11:57:43.514259 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514261 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e" exitCode=0 Sep 16 11:57:43.514259 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514266 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66" exitCode=0 Sep 16 11:57:43.514566 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514311 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884"} Sep 16 11:57:43.514566 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514387 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19"} Sep 16 11:57:43.514566 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514416 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e"} Sep 16 11:57:43.514566 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:43.514440 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66"} Sep 16 11:57:44.012446 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.012385 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.130174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130044 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-web-config\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130174 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130154 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130464 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130197 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130464 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130229 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130464 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130384 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5zwv\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-kube-api-access-h5zwv\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130464 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130429 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-out\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130727 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130469 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-volume\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130727 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130538 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130727 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130594 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-main-db\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130727 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130633 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-cluster-tls-config\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130727 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130678 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-metrics-client-ca\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130727 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130723 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-web\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.130988 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130720 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:44.130988 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.130754 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-tls-assets\") pod \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\" (UID: \"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5\") " Sep 16 11:57:44.131136 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.131105 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:44.131136 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.131115 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 11:57:44.134240 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.134169 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.134523 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.134231 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-out" (OuterVolumeSpecName: "config-out") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 11:57:44.134759 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.134698 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-kube-api-access-h5zwv" (OuterVolumeSpecName: "kube-api-access-h5zwv") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "kube-api-access-h5zwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 11:57:44.134759 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.134734 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.134982 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.134846 2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.134982 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.134880 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.135363 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.135330 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-volume" (OuterVolumeSpecName: "config-volume") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.136936 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.136900 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.137061 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.136934 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 11:57:44.140325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.140090 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.149833 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.149773 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-web-config" (OuterVolumeSpecName: "web-config") pod "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" (UID: "346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:44.236097 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236040 2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236097 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236087 2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-tls-assets\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236097 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236100 2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-web-config\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236097 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236111 2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236124 2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-main-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236136 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zwv\" (UniqueName: \"kubernetes.io/projected/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-kube-api-access-h5zwv\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236144 2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-out\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236153 2551 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-config-volume\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236162 2551 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236171 2551 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-alertmanager-main-db\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236181 2551 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-cluster-tls-config\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.236458 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.236192 2551 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5-metrics-client-ca\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:44.521363 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521324 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161" exitCode=0 Sep 16 11:57:44.521363 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521356 2551 generic.go:356] "Generic (PLEG): container finished" podID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerID="e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534" exitCode=0 Sep 16 11:57:44.521588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521382 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161"} Sep 16 11:57:44.521588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521417 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.521588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521460 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534"} Sep 16 11:57:44.521588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521532 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5","Type":"ContainerDied","Data":"4f261712683d057dfe5da80d41b20ca28f716a7e97b220f92ed4671acdfb6341"} Sep 16 11:57:44.521588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.521540 2551 scope.go:117] "RemoveContainer" containerID="b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884" Sep 16 11:57:44.530859 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.530837 2551 scope.go:117] "RemoveContainer" containerID="ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161" Sep 16 11:57:44.538332 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.538313 2551 scope.go:117] "RemoveContainer" containerID="57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19" Sep 16 11:57:44.546196 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.546165 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:57:44.546916 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.546900 2551 scope.go:117] "RemoveContainer" containerID="e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534" Sep 16 11:57:44.550630 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.550603 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:57:44.555118 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.555097 2551 scope.go:117] "RemoveContainer" containerID="b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e" Sep 16 11:57:44.563142 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.563118 2551 scope.go:117] "RemoveContainer" containerID="7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66" Sep 16 11:57:44.571325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.571301 2551 scope.go:117] "RemoveContainer" containerID="4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e" Sep 16 11:57:44.577778 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.577752 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:57:44.578078 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578066 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="config-reloader" Sep 16 11:57:44.578122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578081 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="config-reloader" Sep 16 11:57:44.578122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578091 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy" Sep 16 11:57:44.578122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578096 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy" Sep 16 11:57:44.578122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578106 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-web" Sep 16 11:57:44.578122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578112 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-web" Sep 16 11:57:44.578122 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578122 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="alertmanager" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578128 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="alertmanager" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578135 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="init-config-reloader" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578142 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="init-config-reloader" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578152 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-metric" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578158 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-metric" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578163 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="prom-label-proxy" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578168 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="prom-label-proxy" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578224 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-web" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578232 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy-metric" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578238 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="kube-rbac-proxy" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578244 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="alertmanager" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578250 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="config-reloader" Sep 16 11:57:44.578306 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.578255 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" containerName="prom-label-proxy" Sep 16 11:57:44.580882 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.580858 2551 scope.go:117] "RemoveContainer" containerID="b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884" Sep 16 11:57:44.580996 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.580901 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.581342 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.581314 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884\": container with ID starting with b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884 not found: ID does not exist" containerID="b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884" Sep 16 11:57:44.581428 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.581356 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884"} err="failed to get container status \"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884\": rpc error: code = NotFound desc = could not find container \"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884\": container with ID starting with b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884 not found: ID does not exist" Sep 16 11:57:44.581428 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.581397 2551 scope.go:117] "RemoveContainer" containerID="ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161" Sep 16 11:57:44.581745 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.581725 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161\": container with ID starting with ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161 not found: ID does not exist" containerID="ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161" Sep 16 11:57:44.581802 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.581751 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161"} err="failed to get container status \"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161\": rpc error: code = NotFound desc = could not find container \"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161\": container with ID starting with ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161 not found: ID does not exist" Sep 16 11:57:44.581802 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.581767 2551 scope.go:117] "RemoveContainer" containerID="57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19" Sep 16 11:57:44.582057 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.582036 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19\": container with ID starting with 57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19 not found: ID does not exist" containerID="57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19" Sep 16 11:57:44.582095 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582067 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19"} err="failed to get container status \"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19\": rpc error: code = NotFound desc = could not find container \"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19\": container with ID starting with 57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19 not found: ID does not exist" Sep 16 11:57:44.582095 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582087 2551 scope.go:117] "RemoveContainer" containerID="e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534" Sep 16 11:57:44.582355 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.582335 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534\": container with ID starting with e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534 not found: ID does not exist" containerID="e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534" Sep 16 11:57:44.582420 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582361 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534"} err="failed to get container status \"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534\": rpc error: code = NotFound desc = could not find container \"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534\": container with ID starting with e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534 not found: ID does not exist" Sep 16 11:57:44.582420 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582376 2551 scope.go:117] "RemoveContainer" containerID="b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e" Sep 16 11:57:44.582679 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.582664 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e\": container with ID starting with b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e not found: ID does not exist" containerID="b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e" Sep 16 11:57:44.582739 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582684 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e"} err="failed to get container status \"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e\": rpc error: code = NotFound desc = could not find container \"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e\": container with ID starting with b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e not found: ID does not exist" Sep 16 11:57:44.582739 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582698 2551 scope.go:117] "RemoveContainer" containerID="7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66" Sep 16 11:57:44.582976 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.582959 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66\": container with ID starting with 7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66 not found: ID does not exist" containerID="7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66" Sep 16 11:57:44.583022 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582978 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66"} err="failed to get container status \"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66\": rpc error: code = NotFound desc = could not find container \"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66\": container with ID starting with 7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66 not found: ID does not exist" Sep 16 11:57:44.583022 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.582990 2551 scope.go:117] "RemoveContainer" containerID="4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e" Sep 16 11:57:44.583216 ip-10-0-140-32 kubenswrapper[2551]: E0916 11:57:44.583197 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e\": container with ID starting with 4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e not found: ID does not exist" containerID="4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e" Sep 16 11:57:44.583278 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583224 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e"} err="failed to get container status \"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e\": rpc error: code = NotFound desc = could not find container \"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e\": container with ID starting with 4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e not found: ID does not exist" Sep 16 11:57:44.583278 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583245 2551 scope.go:117] "RemoveContainer" containerID="b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884" Sep 16 11:57:44.583477 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583461 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884"} err="failed to get container status \"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884\": rpc error: code = NotFound desc = could not find container \"b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884\": container with ID starting with b048f7e13a00c237bdea9083437e86873c8f2071add0cff2be0a7cbd2b9bb884 not found: ID does not exist" Sep 16 11:57:44.583552 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583477 2551 scope.go:117] "RemoveContainer" containerID="ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161" Sep 16 11:57:44.583719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583703 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161"} err="failed to get container status \"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161\": rpc error: code = NotFound desc = could not find container \"ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161\": container with ID starting with ba832e8530e67f3c7299f07e29ecc93a8a8548a3c8c78ba9fbff2dbca76a2161 not found: ID does not exist" Sep 16 11:57:44.583719 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583718 2551 scope.go:117] "RemoveContainer" containerID="57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19" Sep 16 11:57:44.583936 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583918 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19"} err="failed to get container status \"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19\": rpc error: code = NotFound desc = could not find container \"57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19\": container with ID starting with 57d2048a87a3c0d856a857e9ccaf6f36ed96db4b148fe5d26259da8dee615e19 not found: ID does not exist" Sep 16 11:57:44.584006 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.583937 2551 scope.go:117] "RemoveContainer" containerID="e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534" Sep 16 11:57:44.584211 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584194 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534"} err="failed to get container status \"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534\": rpc error: code = NotFound desc = could not find container \"e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534\": container with ID starting with e7dde8b6c90809939f681d79852e20a6faf348e57463e8a78c411de0f912f534 not found: ID does not exist" Sep 16 11:57:44.584266 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584211 2551 scope.go:117] "RemoveContainer" containerID="b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e" Sep 16 11:57:44.584427 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584409 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e"} err="failed to get container status \"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e\": rpc error: code = NotFound desc = could not find container \"b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e\": container with ID starting with b4be137dcd1fb45a3283edc832190deb3d9c487d7056898e14edac6dfa6ffd1e not found: ID does not exist" Sep 16 11:57:44.584518 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584430 2551 scope.go:117] "RemoveContainer" containerID="7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66" Sep 16 11:57:44.584670 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584655 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66"} err="failed to get container status \"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66\": rpc error: code = NotFound desc = could not find container \"7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66\": container with ID starting with 7e4dabb594085c226c69eb6b763192a95db74958199048bd48eedde47b42db66 not found: ID does not exist" Sep 16 11:57:44.584720 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584671 2551 scope.go:117] "RemoveContainer" containerID="4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e" Sep 16 11:57:44.584916 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.584901 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e"} err="failed to get container status \"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e\": rpc error: code = NotFound desc = could not find container \"4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e\": container with ID starting with 4433226cd218ed8f4ef411e87b76f5129b40179496721addf80b475bde8a5e6e not found: ID does not exist" Sep 16 11:57:44.586908 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.586890 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 11:57:44.587331 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.587314 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 11:57:44.587402 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.587315 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 11:57:44.587402 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.587315 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 11:57:44.587553 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.587322 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 11:57:44.588203 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.588187 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 11:57:44.588279 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.588219 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 11:57:44.588279 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.588245 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k6nvs\"" Sep 16 11:57:44.588390 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.588265 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 11:57:44.598935 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.598910 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 11:57:44.635280 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.635244 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:57:44.640104 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640066 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640271 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640117 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2310e3d5-4f76-49af-accb-5abd0da241a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640271 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640176 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640271 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640201 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640271 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640229 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640271 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640251 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640447 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640308 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-web-config\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640447 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640348 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8m4t\" (UniqueName: \"kubernetes.io/projected/2310e3d5-4f76-49af-accb-5abd0da241a0-kube-api-access-v8m4t\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640447 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640384 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2310e3d5-4f76-49af-accb-5abd0da241a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640447 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640402 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640605 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640477 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2310e3d5-4f76-49af-accb-5abd0da241a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640605 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640526 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2310e3d5-4f76-49af-accb-5abd0da241a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.640605 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.640555 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2310e3d5-4f76-49af-accb-5abd0da241a0-config-out\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.717231 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.717177 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5" path="/var/lib/kubelet/pods/346eaa8c-0224-4e16-b1aa-9dfa1f4e03c5/volumes" Sep 16 11:57:44.742007 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.741958 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742007 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742008 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2310e3d5-4f76-49af-accb-5abd0da241a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742292 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742048 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742292 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742073 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742292 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742101 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742292 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742266 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742468 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742339 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-web-config\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742468 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742380 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8m4t\" (UniqueName: \"kubernetes.io/projected/2310e3d5-4f76-49af-accb-5abd0da241a0-kube-api-access-v8m4t\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742468 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742442 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2310e3d5-4f76-49af-accb-5abd0da241a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742729 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742468 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742729 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742556 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2310e3d5-4f76-49af-accb-5abd0da241a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742729 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742575 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2310e3d5-4f76-49af-accb-5abd0da241a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742729 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742589 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2310e3d5-4f76-49af-accb-5abd0da241a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.742729 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.742629 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2310e3d5-4f76-49af-accb-5abd0da241a0-config-out\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745481 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745435 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2310e3d5-4f76-49af-accb-5abd0da241a0-config-out\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745481 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745467 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745481 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745467 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745779 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745533 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745779 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745654 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745779 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745681 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.745779 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.745716 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.746135 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.746112 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2310e3d5-4f76-49af-accb-5abd0da241a0-web-config\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.746237 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.746206 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2310e3d5-4f76-49af-accb-5abd0da241a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.746604 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.746581 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2310e3d5-4f76-49af-accb-5abd0da241a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.747569 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.747549 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2310e3d5-4f76-49af-accb-5abd0da241a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.754071 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.754035 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8m4t\" (UniqueName: \"kubernetes.io/projected/2310e3d5-4f76-49af-accb-5abd0da241a0-kube-api-access-v8m4t\") pod \"alertmanager-main-0\" (UID: \"2310e3d5-4f76-49af-accb-5abd0da241a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:44.890853 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:44.890815 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 11:57:45.170673 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:45.170644 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 11:57:45.172727 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:57:45.172689 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2310e3d5_4f76_49af_accb_5abd0da241a0.slice/crio-54f2736db544b9b6d31e89f6b8f9eeb781e0a963be38a58b3ee52bdecd377640 WatchSource:0}: Error finding container 54f2736db544b9b6d31e89f6b8f9eeb781e0a963be38a58b3ee52bdecd377640: Status 404 returned error can't find the container with id 54f2736db544b9b6d31e89f6b8f9eeb781e0a963be38a58b3ee52bdecd377640 Sep 16 11:57:45.528516 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:45.528405 2551 generic.go:356] "Generic (PLEG): container finished" podID="2310e3d5-4f76-49af-accb-5abd0da241a0" containerID="af5df0b2e98e92f0fcda44d2a01c19952922853d01e40cfeb30edd7963802fcf" exitCode=0 Sep 16 11:57:45.528672 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:45.528486 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerDied","Data":"af5df0b2e98e92f0fcda44d2a01c19952922853d01e40cfeb30edd7963802fcf"} Sep 16 11:57:45.528672 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:45.528589 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"54f2736db544b9b6d31e89f6b8f9eeb781e0a963be38a58b3ee52bdecd377640"} Sep 16 11:57:46.536930 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.536893 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"570d5ba92e9a2840e40b104d17defa69b503e66a512e3b896ca7c44398379dca"} Sep 16 11:57:46.537323 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.536990 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"e8579e14e694717b0b6df4c36c8b531e51a2d2f668da87fd4cf452bb6790c914"} Sep 16 11:57:46.537323 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.537019 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"24c2a380c74329ddc3cddbd350b5a6b42bd5dea5b50499290ab29ab6edf6d66f"} Sep 16 11:57:46.537323 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.537052 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"9290fccfcc8ed62805d8e9fbdb2a305be565b82b96e422bd2c7e72481082dc1c"} Sep 16 11:57:46.537323 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.537087 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"3bdb736856015622758684d0be60d0baa50ef38207797f4c256be29d0125dbfa"} Sep 16 11:57:46.537323 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.537114 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2310e3d5-4f76-49af-accb-5abd0da241a0","Type":"ContainerStarted","Data":"847fdf20f390afea3665e595e9309ccf6c331993f9f872233311fc84267dfd30"} Sep 16 11:57:46.563011 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:46.562949 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.545716439 podStartE2EDuration="2.562930905s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.214466ms" isStatefulPod=false podCreationTimestamp="2026-09-16 11:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:57:46.561844342 +0000 UTC m=+152.347117911" watchObservedRunningTime="2026-09-16 11:57:46.562930905 +0000 UTC m=+152.348204473" Sep 16 11:57:47.092160 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092118 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:57:47.092636 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092556 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="prometheus" containerID="cri-o://1a1a2c679fffca12228e5d94b5446dbc21e4c3b08d7b2aa536d5b62251d3c7f7" gracePeriod=600 Sep 16 11:57:47.092756 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092614 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-web" containerID="cri-o://02ebbc7f9f72f78005532bd6749ec6282cffc3f6aa3b29872a56a045ad7c08c8" gracePeriod=600 Sep 16 11:57:47.092756 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092602 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="config-reloader" containerID="cri-o://d00c8b65ada8afebb195f105bd47f761bce2c782ff534b52938279b58e191f6c" gracePeriod=600 Sep 16 11:57:47.092756 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092592 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="thanos-sidecar" containerID="cri-o://902b65c8eee0a19e2b9b48ee49b32794bc084a396fc829820cb01e1f1fac5401" gracePeriod=600 Sep 16 11:57:47.092902 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092592 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy" containerID="cri-o://e29421072278fe3d8f495eb8f12fb078bedc0ee2c73cde3db5edbc2ae49af54f" gracePeriod=600 Sep 16 11:57:47.092902 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.092602 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c528e384e03f83ad466895009b2ea02c446421a383f4e0dff6af3cc48b0c5617" gracePeriod=600 Sep 16 11:57:47.546162 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546129 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="c528e384e03f83ad466895009b2ea02c446421a383f4e0dff6af3cc48b0c5617" exitCode=0 Sep 16 11:57:47.546162 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546156 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="e29421072278fe3d8f495eb8f12fb078bedc0ee2c73cde3db5edbc2ae49af54f" exitCode=0 Sep 16 11:57:47.546162 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546162 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="02ebbc7f9f72f78005532bd6749ec6282cffc3f6aa3b29872a56a045ad7c08c8" exitCode=0 Sep 16 11:57:47.546162 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546170 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="902b65c8eee0a19e2b9b48ee49b32794bc084a396fc829820cb01e1f1fac5401" exitCode=0 Sep 16 11:57:47.546162 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546179 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="d00c8b65ada8afebb195f105bd47f761bce2c782ff534b52938279b58e191f6c" exitCode=0 Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546185 2551 generic.go:356] "Generic (PLEG): container finished" podID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerID="1a1a2c679fffca12228e5d94b5446dbc21e4c3b08d7b2aa536d5b62251d3c7f7" exitCode=0 Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546194 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"c528e384e03f83ad466895009b2ea02c446421a383f4e0dff6af3cc48b0c5617"} Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546273 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"e29421072278fe3d8f495eb8f12fb078bedc0ee2c73cde3db5edbc2ae49af54f"} Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546317 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"02ebbc7f9f72f78005532bd6749ec6282cffc3f6aa3b29872a56a045ad7c08c8"} Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546357 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"902b65c8eee0a19e2b9b48ee49b32794bc084a396fc829820cb01e1f1fac5401"} Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546391 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"d00c8b65ada8afebb195f105bd47f761bce2c782ff534b52938279b58e191f6c"} Sep 16 11:57:47.546846 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.546431 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"1a1a2c679fffca12228e5d94b5446dbc21e4c3b08d7b2aa536d5b62251d3c7f7"} Sep 16 11:57:47.548935 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.548914 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:47.669595 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669558 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-0\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.669595 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669606 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-metrics-client-ca\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.669881 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669626 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-kubelet-serving-ca-bundle\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.669881 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669648 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-kube-rbac-proxy\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.669881 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669680 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-web-config\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.669881 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669814 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-tls\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.669881 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669863 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-metrics-client-certs\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669899 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config-out\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669934 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-thanos-prometheus-http-client-file\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.669965 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670009 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-tls-assets\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670039 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-2\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670072 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670085 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-trusted-ca-bundle\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670121 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670110 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-1\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670115 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670150 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670182 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-serving-certs-ca-bundle\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670205 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670263 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-db\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670287 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grbmb\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-kube-api-access-grbmb\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670536 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670321 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-grpc-tls\") pod \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\" (UID: \"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf\") " Sep 16 11:57:47.670873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670574 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.670873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670824 2551 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-metrics-client-ca\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.670873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670845 2551 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.670873 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.670861 2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.671133 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.671049 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.671686 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.671532 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.672093 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.671846 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.672543 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.672515 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.672642 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.672541 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.672861 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.672833 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.672958 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.672896 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 11:57:47.673362 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.673331 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.673460 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.673440 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config-out" (OuterVolumeSpecName: "config-out") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 11:57:47.674200 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.674162 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 11:57:47.674200 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.674176 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 11:57:47.674886 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.674856 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.675194 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.675173 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config" (OuterVolumeSpecName: "config") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.675626 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.675601 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-kube-api-access-grbmb" (OuterVolumeSpecName: "kube-api-access-grbmb") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "kube-api-access-grbmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 11:57:47.675626 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.675612 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.675746 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.675626 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.685812 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.685778 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-web-config" (OuterVolumeSpecName: "web-config") pod "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" (UID: "a94d5738-7ffd-465d-9ce7-9ba28dfc5caf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 11:57:47.771311 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771275 2551 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config-out\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771311 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771306 2551 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771311 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771316 2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771325 2551 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-tls-assets\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771334 2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771343 2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771353 2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771362 2551 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771371 2551 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-config\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771380 2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-db\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771389 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grbmb\" (UniqueName: \"kubernetes.io/projected/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-kube-api-access-grbmb\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771401 2551 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-grpc-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771414 2551 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771424 2551 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-kube-rbac-proxy\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771432 2551 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-web-config\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771441 2551 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:47.771638 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:47.771450 2551 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf-secret-metrics-client-certs\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 11:57:48.554649 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.554613 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a94d5738-7ffd-465d-9ce7-9ba28dfc5caf","Type":"ContainerDied","Data":"03d21c19a092ae8aafb47b1796302192c0f3466b9703e997f1c530eea726cd89"} Sep 16 11:57:48.555190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.554690 2551 scope.go:117] "RemoveContainer" containerID="c528e384e03f83ad466895009b2ea02c446421a383f4e0dff6af3cc48b0c5617" Sep 16 11:57:48.555190 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.554630 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.565248 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.565225 2551 scope.go:117] "RemoveContainer" containerID="e29421072278fe3d8f495eb8f12fb078bedc0ee2c73cde3db5edbc2ae49af54f" Sep 16 11:57:48.573792 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.573772 2551 scope.go:117] "RemoveContainer" containerID="02ebbc7f9f72f78005532bd6749ec6282cffc3f6aa3b29872a56a045ad7c08c8" Sep 16 11:57:48.585515 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.583207 2551 scope.go:117] "RemoveContainer" containerID="902b65c8eee0a19e2b9b48ee49b32794bc084a396fc829820cb01e1f1fac5401" Sep 16 11:57:48.585515 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.583419 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:57:48.590681 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.588452 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:57:48.596019 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.595991 2551 scope.go:117] "RemoveContainer" containerID="d00c8b65ada8afebb195f105bd47f761bce2c782ff534b52938279b58e191f6c" Sep 16 11:57:48.603850 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.603828 2551 scope.go:117] "RemoveContainer" containerID="1a1a2c679fffca12228e5d94b5446dbc21e4c3b08d7b2aa536d5b62251d3c7f7" Sep 16 11:57:48.612994 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.612971 2551 scope.go:117] "RemoveContainer" containerID="b13b52905e58367f4c558f4081f19f08d0100b8430b9a49e977eaec5c79f8897" Sep 16 11:57:48.613524 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613484 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:57:48.613879 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613862 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="prometheus" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613882 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="prometheus" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613897 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="init-config-reloader" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613906 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="init-config-reloader" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613915 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-thanos" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613924 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-thanos" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613934 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613942 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613953 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="config-reloader" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613961 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="config-reloader" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613969 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="thanos-sidecar" Sep 16 11:57:48.613975 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613976 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="thanos-sidecar" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.613998 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-web" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614006 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-web" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614099 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="thanos-sidecar" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614113 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-thanos" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614127 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="prometheus" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614137 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy-web" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614147 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="kube-rbac-proxy" Sep 16 11:57:48.614325 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.614158 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" containerName="config-reloader" Sep 16 11:57:48.618307 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.618259 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.622160 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622131 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 11:57:48.622285 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622242 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 11:57:48.622285 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622250 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 11:57:48.622285 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622268 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 11:57:48.622540 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622287 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 11:57:48.622540 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622388 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sjlg7\"" Sep 16 11:57:48.622753 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622579 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 11:57:48.622753 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622690 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 11:57:48.622968 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622950 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 11:57:48.623050 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.622992 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 11:57:48.623050 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.623016 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 11:57:48.623050 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.623031 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 11:57:48.623050 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.623040 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 11:57:48.623270 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.623181 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9pi4ipnvlo06g\"" Sep 16 11:57:48.624991 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.624967 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 11:57:48.627774 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.627672 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 11:57:48.631952 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.631930 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:57:48.680827 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.680758 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.680827 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.680829 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.680867 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.680890 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.680961 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681016 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-config\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681058 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681078 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95b9ef88-fe9f-4414-aa17-8466e27be6b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681105 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681129 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhwd\" (UniqueName: \"kubernetes.io/projected/95b9ef88-fe9f-4414-aa17-8466e27be6b0-kube-api-access-djhwd\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681159 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681163 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95b9ef88-fe9f-4414-aa17-8466e27be6b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681182 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681251 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681295 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681362 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681390 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681433 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681471 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681542 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.681588 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.681562 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.716212 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.716174 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a94d5738-7ffd-465d-9ce7-9ba28dfc5caf" path="/var/lib/kubelet/pods/a94d5738-7ffd-465d-9ce7-9ba28dfc5caf/volumes" Sep 16 11:57:48.783020 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.782983 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783115 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783022 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783115 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783042 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783115 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783072 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783115 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783098 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783294 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783129 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783294 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783156 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783294 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783178 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783294 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783207 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-config\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783294 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783242 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783294 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783267 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95b9ef88-fe9f-4414-aa17-8466e27be6b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783297 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783326 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhwd\" (UniqueName: \"kubernetes.io/projected/95b9ef88-fe9f-4414-aa17-8466e27be6b0-kube-api-access-djhwd\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783360 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95b9ef88-fe9f-4414-aa17-8466e27be6b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783386 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783424 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783458 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783577 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783567 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783932 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783608 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783932 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783644 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.783932 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783910 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.784086 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.783984 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.785081 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.785047 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.785835 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.785804 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.786388 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.786361 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.786490 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.786470 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-web-config\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.786630 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.786579 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95b9ef88-fe9f-4414-aa17-8466e27be6b0-config-out\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.786759 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.786733 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.786886 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.786738 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.786953 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.786889 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.787237 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.787208 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.787903 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.787720 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.787903 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.787789 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.787903 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.787862 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.788878 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.788852 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.788977 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.788938 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-config\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.789220 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.789201 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95b9ef88-fe9f-4414-aa17-8466e27be6b0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.790082 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.790062 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95b9ef88-fe9f-4414-aa17-8466e27be6b0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.790254 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.790235 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95b9ef88-fe9f-4414-aa17-8466e27be6b0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.795621 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.795602 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhwd\" (UniqueName: \"kubernetes.io/projected/95b9ef88-fe9f-4414-aa17-8466e27be6b0-kube-api-access-djhwd\") pod \"prometheus-k8s-0\" (UID: \"95b9ef88-fe9f-4414-aa17-8466e27be6b0\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:48.931706 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:48.931663 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:57:49.218006 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:49.217946 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 11:57:49.220852 ip-10-0-140-32 kubenswrapper[2551]: W0916 11:57:49.220823 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b9ef88_fe9f_4414_aa17_8466e27be6b0.slice/crio-e0278c5af6b0d163f995b9030239c3b758a0988e914368f1606b495e361cbf02 WatchSource:0}: Error finding container e0278c5af6b0d163f995b9030239c3b758a0988e914368f1606b495e361cbf02: Status 404 returned error can't find the container with id e0278c5af6b0d163f995b9030239c3b758a0988e914368f1606b495e361cbf02 Sep 16 11:57:49.562153 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:49.561987 2551 generic.go:356] "Generic (PLEG): container finished" podID="95b9ef88-fe9f-4414-aa17-8466e27be6b0" containerID="e4db22c106619e44570ff2cf3ad735be964752f182a878e8a9bd20e487e07f51" exitCode=0 Sep 16 11:57:49.562153 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:49.562075 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerDied","Data":"e4db22c106619e44570ff2cf3ad735be964752f182a878e8a9bd20e487e07f51"} Sep 16 11:57:49.562642 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:49.562172 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"e0278c5af6b0d163f995b9030239c3b758a0988e914368f1606b495e361cbf02"} Sep 16 11:57:50.572635 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.572602 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"fa1768385f014af016324f917a59c7e4f1343a2a39efaf7efcf8abd4b8c8f339"} Sep 16 11:57:50.573043 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.572697 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"f87f7df49198f83a25fc2d5c721066f8d75f53e75dff14e8e8cdac0c7d4c9cb8"} Sep 16 11:57:50.573043 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.572726 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"354e6d654aa670b360d2be9569cbff1719fd56ea5fcdc3781adec0ecc08f1728"} Sep 16 11:57:50.573043 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.572751 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"b2c0f182f6075b06a38f7846b80b7f21dceb647be49b56cb0efed6d0c7af5da6"} Sep 16 11:57:50.573043 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.572773 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"121a7830e6e601a6dfd126de6a19eea6442d776b68037d4c5568b423dcaa5512"} Sep 16 11:57:50.573043 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.572795 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95b9ef88-fe9f-4414-aa17-8466e27be6b0","Type":"ContainerStarted","Data":"d219db1184dcd173ae4376a14d7f1965bc3ae511aafc5d40cf22fcaee9156818"} Sep 16 11:57:50.608466 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:50.608395 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.58785995 podStartE2EDuration="2.608366647s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.506697ms" isStatefulPod=false podCreationTimestamp="2026-09-16 11:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 11:57:50.602693212 +0000 UTC m=+156.387966807" watchObservedRunningTime="2026-09-16 11:57:50.608366647 +0000 UTC m=+156.393640215" Sep 16 11:57:53.932135 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:57:53.932092 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:58:48.932017 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:58:48.931981 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:58:48.948527 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:58:48.948484 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 11:58:49.868428 ip-10-0-140-32 kubenswrapper[2551]: I0916 11:58:49.868398 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 12:00:14.599717 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:00:14.599681 2551 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 12:02:06.253813 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.253782 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5"] Sep 16 12:02:06.257624 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.257606 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.260001 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.259971 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 16 12:02:06.260001 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.259990 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 16 12:02:06.260210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.260012 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 16 12:02:06.260210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.260059 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 16 12:02:06.260210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.259996 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-x8rmf\"" Sep 16 12:02:06.260210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.260102 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 16 12:02:06.260411 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.260395 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 16 12:02:06.260451 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.260440 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 16 12:02:06.268435 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.268413 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5"] Sep 16 12:02:06.297204 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.297161 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8822d4-7443-4441-a171-d404a07f9260-feast-data\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.297395 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.297232 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-ui-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.297395 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.297283 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-offline-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.297395 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.297348 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.297395 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.297389 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn449\" (UniqueName: \"kubernetes.io/projected/2d8822d4-7443-4441-a171-d404a07f9260-kube-api-access-pn449\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.297573 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.297424 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.397991 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.397944 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8822d4-7443-4441-a171-d404a07f9260-feast-data\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.398195 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.398020 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-ui-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.398195 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.398068 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-offline-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.398195 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.398099 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.398195 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.398133 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn449\" (UniqueName: \"kubernetes.io/projected/2d8822d4-7443-4441-a171-d404a07f9260-kube-api-access-pn449\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.398195 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.398173 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.398453 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:02:06.398268 2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 16 12:02:06.398453 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:02:06.398358 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls podName:2d8822d4-7443-4441-a171-d404a07f9260 nodeName:}" failed. No retries permitted until 2026-09-16 12:02:06.898336346 +0000 UTC m=+412.683609907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls") pod "feast-simple-feast-setup-6d4b9548b7-p4sj5" (UID: "2d8822d4-7443-4441-a171-d404a07f9260") : secret "feast-simple-feast-setup-online-tls" not found Sep 16 12:02:06.398453 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.398416 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8822d4-7443-4441-a171-d404a07f9260-feast-data\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.400695 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.400662 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-offline-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.400695 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.400685 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-ui-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.400837 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.400702 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.409099 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.409070 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn449\" (UniqueName: \"kubernetes.io/projected/2d8822d4-7443-4441-a171-d404a07f9260-kube-api-access-pn449\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.902092 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.902048 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:06.904468 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:06.904444 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-p4sj5\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:07.178466 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:07.178371 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:07.453219 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:07.453142 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5"] Sep 16 12:02:07.456704 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:02:07.456675 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d8822d4_7443_4441_a171_d404a07f9260.slice/crio-35121697e54f775795191821bb8ec094c69fa50d16df57ce3c16e734ee755e4e WatchSource:0}: Error finding container 35121697e54f775795191821bb8ec094c69fa50d16df57ce3c16e734ee755e4e: Status 404 returned error can't find the container with id 35121697e54f775795191821bb8ec094c69fa50d16df57ce3c16e734ee755e4e Sep 16 12:02:07.458585 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:07.458565 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 12:02:07.662506 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:07.662460 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"35121697e54f775795191821bb8ec094c69fa50d16df57ce3c16e734ee755e4e"} Sep 16 12:02:31.780980 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:31.780942 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920"} Sep 16 12:02:36.802270 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:36.802189 2551 generic.go:356] "Generic (PLEG): container finished" podID="2d8822d4-7443-4441-a171-d404a07f9260" containerID="672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920" exitCode=0 Sep 16 12:02:36.802662 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:36.802263 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920"} Sep 16 12:02:37.808771 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:37.808738 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5"} Sep 16 12:02:43.836276 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:43.836190 2551 generic.go:356] "Generic (PLEG): container finished" podID="2d8822d4-7443-4441-a171-d404a07f9260" containerID="b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5" exitCode=0 Sep 16 12:02:43.836276 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:43.836259 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5"} Sep 16 12:02:43.846435 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:43.846413 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 16 12:02:44.846661 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:44.846618 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d"} Sep 16 12:02:44.847146 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:44.846698 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935"} Sep 16 12:02:44.847146 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:44.846719 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622"} Sep 16 12:02:44.847146 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:44.846738 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerStarted","Data":"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886"} Sep 16 12:02:44.875148 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:44.875086 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podStartSLOduration=15.045943696 podStartE2EDuration="38.875065779s" totalImagesPullingTime="23.829122083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:02:44.87235192 +0000 UTC m=+450.657625489" watchObservedRunningTime="2026-09-16 12:02:44.875065779 +0000 UTC m=+450.660339346" Sep 16 12:02:46.179087 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.178911 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:46.179087 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.178977 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:46.179087 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.178998 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:46.179087 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.179015 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:46.181408 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.181222 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Sep 16 12:02:46.181408 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.181254 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 16 12:02:46.181408 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.181259 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Sep 16 12:02:46.181408 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:46.181329 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 16 12:02:47.179096 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:47.179042 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:47.179096 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:47.179117 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:47.179717 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:47.179138 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:47.179717 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:47.179154 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:49.179706 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:49.179659 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 16 12:02:49.180329 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:49.179658 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Sep 16 12:02:49.180475 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:49.179672 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 16 12:02:49.180475 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:49.179681 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Sep 16 12:02:52.179675 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:52.179575 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Sep 16 12:02:52.180133 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:52.179575 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.21:6571: connect: connection refused" Sep 16 12:02:52.180133 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:52.179575 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.21:8816: connect: connection refused" Sep 16 12:02:52.180133 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:52.179837 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 16 12:02:55.179888 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.179843 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.180452 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.180006 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.180452 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.180354 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.180977 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.180950 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.181096 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.181026 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.181096 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.181067 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.187130 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.187104 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:02:55.927881 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:02:55.927852 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:03:58.584446 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.584409 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5"] Sep 16 12:03:58.584974 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.584733 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" containerID="cri-o://9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886" gracePeriod=30 Sep 16 12:03:58.584974 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.584785 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" containerID="cri-o://6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935" gracePeriod=30 Sep 16 12:03:58.584974 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.584806 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" containerID="cri-o://6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d" gracePeriod=30 Sep 16 12:03:58.584974 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.584812 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" containerID="cri-o://7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622" gracePeriod=30 Sep 16 12:03:58.940873 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.940835 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8"] Sep 16 12:03:58.945615 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.945587 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:58.947930 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.947902 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jmg5z\"" Sep 16 12:03:58.957133 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:58.957066 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8"] Sep 16 12:03:59.036806 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.036743 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.036806 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.036809 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.037089 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.036889 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-ui-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.037089 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.036919 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/10440004-48fd-49ac-b68b-6f9f202eecd1-feast-data\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.037089 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.036997 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-offline-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.037242 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.037090 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sgm\" (UniqueName: \"kubernetes.io/projected/10440004-48fd-49ac-b68b-6f9f202eecd1-kube-api-access-p5sgm\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.138648 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.138607 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-offline-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.138867 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.138687 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sgm\" (UniqueName: \"kubernetes.io/projected/10440004-48fd-49ac-b68b-6f9f202eecd1-kube-api-access-p5sgm\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.138867 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.138734 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.138867 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.138757 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.138867 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.138809 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-ui-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.138867 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.138832 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/10440004-48fd-49ac-b68b-6f9f202eecd1-feast-data\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.139112 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:03:59.138879 2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 16 12:03:59.139112 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:03:59.138945 2551 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 16 12:03:59.139112 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:03:59.138974 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls podName:10440004-48fd-49ac-b68b-6f9f202eecd1 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.638938293 +0000 UTC m=+525.424211858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls") pod "feast-simple-feast-setup-6d4b9548b7-rr5t8" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1") : secret "feast-simple-feast-setup-registry-tls" not found Sep 16 12:03:59.139112 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:03:59.139010 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls podName:10440004-48fd-49ac-b68b-6f9f202eecd1 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:59.638992421 +0000 UTC m=+525.424265983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls") pod "feast-simple-feast-setup-6d4b9548b7-rr5t8" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1") : secret "feast-simple-feast-setup-online-tls" not found Sep 16 12:03:59.139321 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.139274 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/10440004-48fd-49ac-b68b-6f9f202eecd1-feast-data\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.141415 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.141376 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-offline-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.141415 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.141395 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-ui-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.150575 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.150546 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sgm\" (UniqueName: \"kubernetes.io/projected/10440004-48fd-49ac-b68b-6f9f202eecd1-kube-api-access-p5sgm\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.644138 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.644108 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.644625 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.644142 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.646577 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.646552 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.646688 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.646651 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls\") pod \"feast-simple-feast-setup-6d4b9548b7-rr5t8\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:03:59.857588 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:03:59.857553 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:00.138741 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.138715 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8"] Sep 16 12:04:00.140364 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:04:00.140341 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10440004_48fd_49ac_b68b_6f9f202eecd1.slice/crio-eb059929564bb6dfa61ae0756867031d939f437bcc74bab622ac5c4d201f05ec WatchSource:0}: Error finding container eb059929564bb6dfa61ae0756867031d939f437bcc74bab622ac5c4d201f05ec: Status 404 returned error can't find the container with id eb059929564bb6dfa61ae0756867031d939f437bcc74bab622ac5c4d201f05ec Sep 16 12:04:00.204112 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.204084 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358"} Sep 16 12:04:00.204261 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.204133 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"eb059929564bb6dfa61ae0756867031d939f437bcc74bab622ac5c4d201f05ec"} Sep 16 12:04:00.207210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.207183 2551 generic.go:356] "Generic (PLEG): container finished" podID="2d8822d4-7443-4441-a171-d404a07f9260" containerID="6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d" exitCode=0 Sep 16 12:04:00.207210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.207204 2551 generic.go:356] "Generic (PLEG): container finished" podID="2d8822d4-7443-4441-a171-d404a07f9260" containerID="7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622" exitCode=0 Sep 16 12:04:00.207360 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.207223 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d"} Sep 16 12:04:00.207360 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:00.207251 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622"} Sep 16 12:04:05.181452 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:05.181396 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Sep 16 12:04:05.925268 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:05.925226 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 16 12:04:06.237222 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:06.237129 2551 generic.go:356] "Generic (PLEG): container finished" podID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerID="f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358" exitCode=0 Sep 16 12:04:06.237222 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:06.237206 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358"} Sep 16 12:04:07.244245 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:07.244206 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454"} Sep 16 12:04:13.270462 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:13.270428 2551 generic.go:356] "Generic (PLEG): container finished" podID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerID="72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454" exitCode=0 Sep 16 12:04:13.270861 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:13.270514 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454"} Sep 16 12:04:14.282185 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.282145 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8"} Sep 16 12:04:14.282706 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.282213 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40"} Sep 16 12:04:14.282706 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.282235 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525"} Sep 16 12:04:14.282706 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.282254 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerStarted","Data":"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068"} Sep 16 12:04:14.311398 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.311161 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podStartSLOduration=16.311134109 podStartE2EDuration="16.311134109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:14.308465371 +0000 UTC m=+540.093738939" watchObservedRunningTime="2026-09-16 12:04:14.311134109 +0000 UTC m=+540.096407677" Sep 16 12:04:14.858036 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.858003 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:14.858315 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.858293 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:14.858451 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.858326 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:14.858451 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.858347 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:14.860147 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.860106 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 16 12:04:14.860267 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.860162 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 16 12:04:14.860267 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.860248 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Sep 16 12:04:14.860386 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:14.860340 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Sep 16 12:04:15.181535 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:15.181453 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Sep 16 12:04:15.925700 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:15.925637 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 16 12:04:17.858331 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:17.858278 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Sep 16 12:04:17.858858 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:17.858278 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Sep 16 12:04:17.858858 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:17.858422 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 16 12:04:17.858858 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:17.858761 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 16 12:04:19.858772 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:19.858687 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:19.858772 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:19.858752 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:19.858772 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:19.858772 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:19.859374 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:19.858790 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:20.858225 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:20.858173 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 16 12:04:20.858438 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:20.858173 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.22:6571: connect: connection refused" Sep 16 12:04:20.858438 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:20.858173 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.22:8816: connect: connection refused" Sep 16 12:04:20.858438 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:20.858190 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 16 12:04:23.858809 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.858774 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:23.859229 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.858912 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:23.859229 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.859164 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:23.860011 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.859989 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:23.860138 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.860055 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:23.860138 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.860117 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:23.864965 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:23.864941 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:24.345675 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:24.345647 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:04:25.180898 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:25.180856 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.21:8443: connect: connection refused" Sep 16 12:04:25.181361 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:25.180991 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:04:25.925165 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:25.925126 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" probeResult="failure" output="Get \"https://10.134.0.21:6567/health\": dial tcp 10.134.0.21:6567: connect: connection refused" Sep 16 12:04:25.925321 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:25.925244 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:04:29.244253 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.244224 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:04:29.323771 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.323728 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-registry-tls\") pod \"2d8822d4-7443-4441-a171-d404a07f9260\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " Sep 16 12:04:29.323934 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.323803 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8822d4-7443-4441-a171-d404a07f9260-feast-data\") pod \"2d8822d4-7443-4441-a171-d404a07f9260\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " Sep 16 12:04:29.323934 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.323903 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls\") pod \"2d8822d4-7443-4441-a171-d404a07f9260\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " Sep 16 12:04:29.324037 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.323957 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-ui-tls\") pod \"2d8822d4-7443-4441-a171-d404a07f9260\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " Sep 16 12:04:29.324037 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.323982 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-offline-tls\") pod \"2d8822d4-7443-4441-a171-d404a07f9260\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " Sep 16 12:04:29.324037 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.324000 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn449\" (UniqueName: \"kubernetes.io/projected/2d8822d4-7443-4441-a171-d404a07f9260-kube-api-access-pn449\") pod \"2d8822d4-7443-4441-a171-d404a07f9260\" (UID: \"2d8822d4-7443-4441-a171-d404a07f9260\") " Sep 16 12:04:29.324415 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.324383 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d8822d4-7443-4441-a171-d404a07f9260-feast-data" (OuterVolumeSpecName: "feast-data") pod "2d8822d4-7443-4441-a171-d404a07f9260" (UID: "2d8822d4-7443-4441-a171-d404a07f9260"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:04:29.326239 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.326196 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2d8822d4-7443-4441-a171-d404a07f9260" (UID: "2d8822d4-7443-4441-a171-d404a07f9260"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:04:29.326376 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.326257 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls" (OuterVolumeSpecName: "online-tls") pod "2d8822d4-7443-4441-a171-d404a07f9260" (UID: "2d8822d4-7443-4441-a171-d404a07f9260"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:04:29.326376 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.326289 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2d8822d4-7443-4441-a171-d404a07f9260" (UID: "2d8822d4-7443-4441-a171-d404a07f9260"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:04:29.326465 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.326446 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2d8822d4-7443-4441-a171-d404a07f9260" (UID: "2d8822d4-7443-4441-a171-d404a07f9260"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:04:29.326534 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.326463 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8822d4-7443-4441-a171-d404a07f9260-kube-api-access-pn449" (OuterVolumeSpecName: "kube-api-access-pn449") pod "2d8822d4-7443-4441-a171-d404a07f9260" (UID: "2d8822d4-7443-4441-a171-d404a07f9260"). InnerVolumeSpecName "kube-api-access-pn449". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:04:29.365410 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365317 2551 generic.go:356] "Generic (PLEG): container finished" podID="2d8822d4-7443-4441-a171-d404a07f9260" containerID="6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935" exitCode=137 Sep 16 12:04:29.365410 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365353 2551 generic.go:356] "Generic (PLEG): container finished" podID="2d8822d4-7443-4441-a171-d404a07f9260" containerID="9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886" exitCode=137 Sep 16 12:04:29.365410 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365399 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" Sep 16 12:04:29.365692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365406 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935"} Sep 16 12:04:29.365692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365456 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886"} Sep 16 12:04:29.365692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365471 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5" event={"ID":"2d8822d4-7443-4441-a171-d404a07f9260","Type":"ContainerDied","Data":"35121697e54f775795191821bb8ec094c69fa50d16df57ce3c16e734ee755e4e"} Sep 16 12:04:29.365692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.365489 2551 scope.go:117] "RemoveContainer" containerID="6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d" Sep 16 12:04:29.374790 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.374764 2551 scope.go:117] "RemoveContainer" containerID="6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935" Sep 16 12:04:29.383584 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.383570 2551 scope.go:117] "RemoveContainer" containerID="7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622" Sep 16 12:04:29.388890 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.388852 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5"] Sep 16 12:04:29.391778 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.391752 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-p4sj5"] Sep 16 12:04:29.393096 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.393071 2551 scope.go:117] "RemoveContainer" containerID="9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886" Sep 16 12:04:29.401607 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.401587 2551 scope.go:117] "RemoveContainer" containerID="b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5" Sep 16 12:04:29.409292 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.409274 2551 scope.go:117] "RemoveContainer" containerID="672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920" Sep 16 12:04:29.416841 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.416822 2551 scope.go:117] "RemoveContainer" containerID="6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d" Sep 16 12:04:29.417161 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:04:29.417143 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d\": container with ID starting with 6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d not found: ID does not exist" containerID="6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d" Sep 16 12:04:29.417206 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.417171 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d"} err="failed to get container status \"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d\": rpc error: code = NotFound desc = could not find container \"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d\": container with ID starting with 6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d not found: ID does not exist" Sep 16 12:04:29.417206 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.417193 2551 scope.go:117] "RemoveContainer" containerID="6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935" Sep 16 12:04:29.417431 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:04:29.417416 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935\": container with ID starting with 6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935 not found: ID does not exist" containerID="6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935" Sep 16 12:04:29.417481 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.417436 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935"} err="failed to get container status \"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935\": rpc error: code = NotFound desc = could not find container \"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935\": container with ID starting with 6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935 not found: ID does not exist" Sep 16 12:04:29.417481 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.417451 2551 scope.go:117] "RemoveContainer" containerID="7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622" Sep 16 12:04:29.417739 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:04:29.417723 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622\": container with ID starting with 7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622 not found: ID does not exist" containerID="7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622" Sep 16 12:04:29.417790 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.417743 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622"} err="failed to get container status \"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622\": rpc error: code = NotFound desc = could not find container \"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622\": container with ID starting with 7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622 not found: ID does not exist" Sep 16 12:04:29.417790 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.417757 2551 scope.go:117] "RemoveContainer" containerID="9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886" Sep 16 12:04:29.418001 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:04:29.417988 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886\": container with ID starting with 9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886 not found: ID does not exist" containerID="9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886" Sep 16 12:04:29.418041 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418004 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886"} err="failed to get container status \"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886\": rpc error: code = NotFound desc = could not find container \"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886\": container with ID starting with 9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886 not found: ID does not exist" Sep 16 12:04:29.418041 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418015 2551 scope.go:117] "RemoveContainer" containerID="b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5" Sep 16 12:04:29.418211 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:04:29.418198 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5\": container with ID starting with b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5 not found: ID does not exist" containerID="b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5" Sep 16 12:04:29.418305 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418212 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5"} err="failed to get container status \"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5\": rpc error: code = NotFound desc = could not find container \"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5\": container with ID starting with b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5 not found: ID does not exist" Sep 16 12:04:29.418305 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418223 2551 scope.go:117] "RemoveContainer" containerID="672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920" Sep 16 12:04:29.418483 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:04:29.418460 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920\": container with ID starting with 672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920 not found: ID does not exist" containerID="672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920" Sep 16 12:04:29.418554 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418489 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920"} err="failed to get container status \"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920\": rpc error: code = NotFound desc = could not find container \"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920\": container with ID starting with 672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920 not found: ID does not exist" Sep 16 12:04:29.418554 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418534 2551 scope.go:117] "RemoveContainer" containerID="6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d" Sep 16 12:04:29.418794 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418777 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d"} err="failed to get container status \"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d\": rpc error: code = NotFound desc = could not find container \"6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d\": container with ID starting with 6fc23fc87c611726263b2db69d668d8dfcc42fc18aa1af6c48ae16bcec17b34d not found: ID does not exist" Sep 16 12:04:29.418840 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418794 2551 scope.go:117] "RemoveContainer" containerID="6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935" Sep 16 12:04:29.419004 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.418983 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935"} err="failed to get container status \"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935\": rpc error: code = NotFound desc = could not find container \"6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935\": container with ID starting with 6601e14c396661da930798f6c2cb5864e7eb0c70177c93f99aef3f7c71835935 not found: ID does not exist" Sep 16 12:04:29.419070 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419006 2551 scope.go:117] "RemoveContainer" containerID="7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622" Sep 16 12:04:29.419280 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419262 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622"} err="failed to get container status \"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622\": rpc error: code = NotFound desc = could not find container \"7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622\": container with ID starting with 7a2382a6a424e361082a164abf49d6993d1774b485839edfb0cfb3e3755c8622 not found: ID does not exist" Sep 16 12:04:29.419333 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419280 2551 scope.go:117] "RemoveContainer" containerID="9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886" Sep 16 12:04:29.419521 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419486 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886"} err="failed to get container status \"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886\": rpc error: code = NotFound desc = could not find container \"9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886\": container with ID starting with 9289e95ceba852068578db2f6055bee58558ac0ace8cd920c7ca9d239bf74886 not found: ID does not exist" Sep 16 12:04:29.419588 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419523 2551 scope.go:117] "RemoveContainer" containerID="b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5" Sep 16 12:04:29.419746 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419730 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5"} err="failed to get container status \"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5\": rpc error: code = NotFound desc = could not find container \"b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5\": container with ID starting with b500db1f6dda1629f8377d2a37a58e0ef32ea8f2c6f44c1183bfdb9f0f5c5fc5 not found: ID does not exist" Sep 16 12:04:29.419746 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419746 2551 scope.go:117] "RemoveContainer" containerID="672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920" Sep 16 12:04:29.419956 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.419940 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920"} err="failed to get container status \"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920\": rpc error: code = NotFound desc = could not find container \"672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920\": container with ID starting with 672b9a18407f792f67acc7b90e84b16a1f7463cad826f9c65ae11560bf382920 not found: ID does not exist" Sep 16 12:04:29.425392 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.425368 2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-online-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:04:29.425392 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.425388 2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-ui-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:04:29.425392 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.425396 2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-offline-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:04:29.425626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.425406 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn449\" (UniqueName: \"kubernetes.io/projected/2d8822d4-7443-4441-a171-d404a07f9260-kube-api-access-pn449\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:04:29.425626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.425414 2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/2d8822d4-7443-4441-a171-d404a07f9260-registry-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:04:29.425626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:29.425422 2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2d8822d4-7443-4441-a171-d404a07f9260-feast-data\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:04:30.716077 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:04:30.716042 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d8822d4-7443-4441-a171-d404a07f9260" path="/var/lib/kubelet/pods/2d8822d4-7443-4441-a171-d404a07f9260/volumes" Sep 16 12:05:26.005772 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.005732 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv"] Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006063 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006073 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006091 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="feast-init" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006097 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="feast-init" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006105 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006111 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006122 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006128 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006135 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006140 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006147 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="feast-apply" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006153 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="feast-apply" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006205 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="online" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006213 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="registry" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006219 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="offline" Sep 16 12:05:26.006235 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.006226 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8822d4-7443-4441-a171-d404a07f9260" containerName="ui" Sep 16 12:05:26.011142 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.011115 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.013351 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013320 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 16 12:05:26.013568 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013348 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 16 12:05:26.013568 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013348 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 16 12:05:26.013568 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013439 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 16 12:05:26.013737 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013588 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-j5m4p\"" Sep 16 12:05:26.013737 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013679 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 16 12:05:26.013737 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013692 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 16 12:05:26.013737 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.013700 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 16 12:05:26.024383 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.024350 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv"] Sep 16 12:05:26.134906 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.134864 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-ui-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.134906 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.134909 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-feast-data\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.135134 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.134943 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-offline-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.135134 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.134960 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9q4\" (UniqueName: \"kubernetes.io/projected/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-kube-api-access-rx9q4\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.135134 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.135051 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-registry-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.135134 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.135084 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236428 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.236383 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-offline-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236428 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.236429 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9q4\" (UniqueName: \"kubernetes.io/projected/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-kube-api-access-rx9q4\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236699 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.236477 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-registry-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236699 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.236533 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236699 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.236623 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-ui-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236699 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.236650 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-feast-data\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.236917 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:05:26.236706 2551 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 16 12:05:26.236917 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:05:26.236797 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls podName:24b03bec-e7d0-4da5-940a-e2ba5ee0235a nodeName:}" failed. No retries permitted until 2026-09-16 12:05:26.736773041 +0000 UTC m=+612.522046593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls") pod "feast-simple-feast-remote-setup-b469885bc-qkrrv" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 16 12:05:26.237104 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.237083 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-feast-data\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.237340 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.237318 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-registry-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.239062 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.239037 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-offline-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.239170 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.239085 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-ui-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.246877 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.246845 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9q4\" (UniqueName: \"kubernetes.io/projected/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-kube-api-access-rx9q4\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.740896 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.740827 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.743970 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.743937 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls\") pod \"feast-simple-feast-remote-setup-b469885bc-qkrrv\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:26.921626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:26.921576 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:05:27.200177 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:27.200145 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv"] Sep 16 12:05:27.615269 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:27.615177 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerStarted","Data":"38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7"} Sep 16 12:05:27.615269 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:27.615227 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerStarted","Data":"70db8eacb481922231d6a8e183c7e1ccd7700096fd02f64aec5d6dc6cd682d95"} Sep 16 12:05:33.650649 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:33.650612 2551 generic.go:356] "Generic (PLEG): container finished" podID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerID="38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7" exitCode=0 Sep 16 12:05:33.651202 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:33.650693 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerDied","Data":"38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7"} Sep 16 12:05:34.658467 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:34.658429 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerStarted","Data":"89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493"} Sep 16 12:05:59.769670 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:59.769633 2551 generic.go:356] "Generic (PLEG): container finished" podID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerID="89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493" exitCode=0 Sep 16 12:05:59.770089 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:05:59.769706 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerDied","Data":"89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493"} Sep 16 12:06:00.779937 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:00.779902 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerStarted","Data":"af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757"} Sep 16 12:06:00.780326 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:00.779962 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerStarted","Data":"690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0"} Sep 16 12:06:00.780326 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:00.779979 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerStarted","Data":"0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9"} Sep 16 12:06:00.806683 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:00.806617 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podStartSLOduration=35.80659451 podStartE2EDuration="35.80659451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:06:00.804789338 +0000 UTC m=+646.590062919" watchObservedRunningTime="2026-09-16 12:06:00.80659451 +0000 UTC m=+646.591868078" Sep 16 12:06:02.922919 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:02.922831 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:02.923608 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:02.922962 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:02.923608 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:02.922991 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:02.925249 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:02.925187 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 16 12:06:02.925427 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:02.925235 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 16 12:06:02.925427 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:02.925209 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 16 12:06:05.922797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:05.922742 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 16 12:06:05.922797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:05.922781 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 16 12:06:05.923345 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:05.922758 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.23:8816: connect: connection refused" Sep 16 12:06:06.922732 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:06.922686 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:06.922929 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:06.922763 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:06.922929 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:06.922790 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:08.922861 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:08.922815 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:08.923358 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:08.923105 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:08.923703 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:08.923679 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:08.923845 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:08.923747 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:08.949727 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:08.949695 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:06:09.847305 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:06:09.847270 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:07:12.446813 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.446724 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv"] Sep 16 12:07:12.447338 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.447013 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" containerID="cri-o://0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9" gracePeriod=30 Sep 16 12:07:12.447338 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.447095 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" containerID="cri-o://af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757" gracePeriod=30 Sep 16 12:07:12.447338 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.447108 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="offline" containerID="cri-o://690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0" gracePeriod=30 Sep 16 12:07:12.575270 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.575235 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8"] Sep 16 12:07:12.575626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.575564 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" containerID="cri-o://40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068" gracePeriod=30 Sep 16 12:07:12.575626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.575605 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" containerID="cri-o://efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525" gracePeriod=30 Sep 16 12:07:12.575904 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.575605 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" containerID="cri-o://026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40" gracePeriod=30 Sep 16 12:07:12.575904 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:12.575605 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" containerID="cri-o://63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8" gracePeriod=30 Sep 16 12:07:13.860720 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:13.860672 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 16 12:07:14.110823 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.110729 2551 generic.go:356] "Generic (PLEG): container finished" podID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerID="af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757" exitCode=0 Sep 16 12:07:14.110823 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.110760 2551 generic.go:356] "Generic (PLEG): container finished" podID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerID="0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9" exitCode=0 Sep 16 12:07:14.110823 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.110797 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerDied","Data":"af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757"} Sep 16 12:07:14.111097 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.110838 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerDied","Data":"0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9"} Sep 16 12:07:14.113772 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.113746 2551 generic.go:356] "Generic (PLEG): container finished" podID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerID="63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8" exitCode=0 Sep 16 12:07:14.113772 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.113767 2551 generic.go:356] "Generic (PLEG): container finished" podID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerID="efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525" exitCode=0 Sep 16 12:07:14.113913 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.113812 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8"} Sep 16 12:07:14.113913 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.113845 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525"} Sep 16 12:07:14.343653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:14.343609 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 16 12:07:18.923925 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:18.923880 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 16 12:07:19.828695 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:19.828650 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 16 12:07:23.860137 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:23.860091 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 16 12:07:24.342885 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:24.342841 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 16 12:07:28.923693 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:28.923647 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 16 12:07:29.828747 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:29.828704 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 16 12:07:33.859750 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:33.859701 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.22:8443: connect: connection refused" Sep 16 12:07:33.860214 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:33.859892 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:07:34.343400 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:34.343355 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.22:6567/health\": dial tcp 10.134.0.22:6567: connect: connection refused" Sep 16 12:07:34.346199 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:34.344109 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:07:38.923851 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:38.923803 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.23:8443: connect: connection refused" Sep 16 12:07:38.924301 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:38.923963 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:07:39.828797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:39.828751 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.23:6567/health\": dial tcp 10.134.0.23:6567: connect: connection refused" Sep 16 12:07:39.829012 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:39.828879 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:07:43.099338 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.099308 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:07:43.226713 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.226689 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:07:43.236898 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.236869 2551 generic.go:356] "Generic (PLEG): container finished" podID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerID="690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0" exitCode=137 Sep 16 12:07:43.237066 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.236929 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" Sep 16 12:07:43.237066 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.236932 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerDied","Data":"690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0"} Sep 16 12:07:43.237066 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.236991 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv" event={"ID":"24b03bec-e7d0-4da5-940a-e2ba5ee0235a","Type":"ContainerDied","Data":"70db8eacb481922231d6a8e183c7e1ccd7700096fd02f64aec5d6dc6cd682d95"} Sep 16 12:07:43.237066 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.237017 2551 scope.go:117] "RemoveContainer" containerID="af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757" Sep 16 12:07:43.239813 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.239788 2551 generic.go:356] "Generic (PLEG): container finished" podID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerID="026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40" exitCode=137 Sep 16 12:07:43.239813 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.239811 2551 generic.go:356] "Generic (PLEG): container finished" podID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerID="40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068" exitCode=137 Sep 16 12:07:43.239963 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.239825 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40"} Sep 16 12:07:43.239963 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.239869 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" Sep 16 12:07:43.239963 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.239869 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068"} Sep 16 12:07:43.240058 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.239965 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8" event={"ID":"10440004-48fd-49ac-b68b-6f9f202eecd1","Type":"ContainerDied","Data":"eb059929564bb6dfa61ae0756867031d939f437bcc74bab622ac5c4d201f05ec"} Sep 16 12:07:43.248333 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.248312 2551 scope.go:117] "RemoveContainer" containerID="690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0" Sep 16 12:07:43.256482 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.256462 2551 scope.go:117] "RemoveContainer" containerID="0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9" Sep 16 12:07:43.264603 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.264581 2551 scope.go:117] "RemoveContainer" containerID="89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493" Sep 16 12:07:43.274113 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.274091 2551 scope.go:117] "RemoveContainer" containerID="38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7" Sep 16 12:07:43.282516 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.282482 2551 scope.go:117] "RemoveContainer" containerID="af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757" Sep 16 12:07:43.282866 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.282847 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757\": container with ID starting with af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757 not found: ID does not exist" containerID="af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757" Sep 16 12:07:43.282944 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.282880 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757"} err="failed to get container status \"af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757\": rpc error: code = NotFound desc = could not find container \"af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757\": container with ID starting with af8eb07e9aa00f39b1ef03618d30462e4193780249759a32190b7f13ab8a5757 not found: ID does not exist" Sep 16 12:07:43.282944 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.282903 2551 scope.go:117] "RemoveContainer" containerID="690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0" Sep 16 12:07:43.283202 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.283184 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0\": container with ID starting with 690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0 not found: ID does not exist" containerID="690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0" Sep 16 12:07:43.283249 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.283209 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0"} err="failed to get container status \"690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0\": rpc error: code = NotFound desc = could not find container \"690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0\": container with ID starting with 690be0fa315a32cdddbc6258f136daaeed6b2d699a015fa92c345bd5e081a8e0 not found: ID does not exist" Sep 16 12:07:43.283249 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.283225 2551 scope.go:117] "RemoveContainer" containerID="0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9" Sep 16 12:07:43.283628 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.283604 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9\": container with ID starting with 0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9 not found: ID does not exist" containerID="0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9" Sep 16 12:07:43.283714 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.283632 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9"} err="failed to get container status \"0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9\": rpc error: code = NotFound desc = could not find container \"0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9\": container with ID starting with 0acdadbb773c2ac75427802ccde042aecca07fce2e1b6b83a2677e0d2684f7e9 not found: ID does not exist" Sep 16 12:07:43.283714 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.283648 2551 scope.go:117] "RemoveContainer" containerID="89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493" Sep 16 12:07:43.283900 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.283885 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493\": container with ID starting with 89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493 not found: ID does not exist" containerID="89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493" Sep 16 12:07:43.283948 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.283904 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493"} err="failed to get container status \"89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493\": rpc error: code = NotFound desc = could not find container \"89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493\": container with ID starting with 89deb040f680f2c9b913f280dbe8424f3f1e8ef4cc123e23fd6cef9b483cd493 not found: ID does not exist" Sep 16 12:07:43.283948 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.283917 2551 scope.go:117] "RemoveContainer" containerID="38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7" Sep 16 12:07:43.284117 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.284098 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7\": container with ID starting with 38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7 not found: ID does not exist" containerID="38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7" Sep 16 12:07:43.284175 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.284118 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7"} err="failed to get container status \"38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7\": rpc error: code = NotFound desc = could not find container \"38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7\": container with ID starting with 38baf89299bf33e65df9ef463cd3f73334f540724665dc3d97027b362ef59fe7 not found: ID does not exist" Sep 16 12:07:43.284175 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.284133 2551 scope.go:117] "RemoveContainer" containerID="63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8" Sep 16 12:07:43.287310 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287289 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-registry-tls\") pod \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " Sep 16 12:07:43.287478 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287462 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls\") pod \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " Sep 16 12:07:43.287542 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287529 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-feast-data\") pod \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " Sep 16 12:07:43.287604 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287586 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-ui-tls\") pod \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " Sep 16 12:07:43.287675 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287610 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "24b03bec-e7d0-4da5-940a-e2ba5ee0235a" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 12:07:43.287675 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287631 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-offline-tls\") pod \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " Sep 16 12:07:43.287760 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287679 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx9q4\" (UniqueName: \"kubernetes.io/projected/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-kube-api-access-rx9q4\") pod \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\" (UID: \"24b03bec-e7d0-4da5-940a-e2ba5ee0235a\") " Sep 16 12:07:43.287963 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.287942 2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-registry-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.288059 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.288036 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-feast-data" (OuterVolumeSpecName: "feast-data") pod "24b03bec-e7d0-4da5-940a-e2ba5ee0235a" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:07:43.289819 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.289781 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls" (OuterVolumeSpecName: "online-tls") pod "24b03bec-e7d0-4da5-940a-e2ba5ee0235a" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.289905 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.289875 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "24b03bec-e7d0-4da5-940a-e2ba5ee0235a" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.289947 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.289915 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "24b03bec-e7d0-4da5-940a-e2ba5ee0235a" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.289981 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.289944 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-kube-api-access-rx9q4" (OuterVolumeSpecName: "kube-api-access-rx9q4") pod "24b03bec-e7d0-4da5-940a-e2ba5ee0235a" (UID: "24b03bec-e7d0-4da5-940a-e2ba5ee0235a"). InnerVolumeSpecName "kube-api-access-rx9q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:07:43.292770 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.292749 2551 scope.go:117] "RemoveContainer" containerID="026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40" Sep 16 12:07:43.300838 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.300817 2551 scope.go:117] "RemoveContainer" containerID="efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525" Sep 16 12:07:43.308848 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.308829 2551 scope.go:117] "RemoveContainer" containerID="40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068" Sep 16 12:07:43.317019 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.316993 2551 scope.go:117] "RemoveContainer" containerID="72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454" Sep 16 12:07:43.325050 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.325029 2551 scope.go:117] "RemoveContainer" containerID="f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358" Sep 16 12:07:43.332881 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.332857 2551 scope.go:117] "RemoveContainer" containerID="63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8" Sep 16 12:07:43.333254 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.333229 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8\": container with ID starting with 63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8 not found: ID does not exist" containerID="63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8" Sep 16 12:07:43.333330 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.333266 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8"} err="failed to get container status \"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8\": rpc error: code = NotFound desc = could not find container \"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8\": container with ID starting with 63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8 not found: ID does not exist" Sep 16 12:07:43.333330 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.333287 2551 scope.go:117] "RemoveContainer" containerID="026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40" Sep 16 12:07:43.333624 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.333607 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40\": container with ID starting with 026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40 not found: ID does not exist" containerID="026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40" Sep 16 12:07:43.333700 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.333630 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40"} err="failed to get container status \"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40\": rpc error: code = NotFound desc = could not find container \"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40\": container with ID starting with 026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40 not found: ID does not exist" Sep 16 12:07:43.333700 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.333645 2551 scope.go:117] "RemoveContainer" containerID="efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525" Sep 16 12:07:43.333900 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.333882 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525\": container with ID starting with efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525 not found: ID does not exist" containerID="efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525" Sep 16 12:07:43.333940 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.333909 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525"} err="failed to get container status \"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525\": rpc error: code = NotFound desc = could not find container \"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525\": container with ID starting with efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525 not found: ID does not exist" Sep 16 12:07:43.333940 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.333926 2551 scope.go:117] "RemoveContainer" containerID="40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068" Sep 16 12:07:43.334166 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.334147 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068\": container with ID starting with 40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068 not found: ID does not exist" containerID="40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068" Sep 16 12:07:43.334239 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.334175 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068"} err="failed to get container status \"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068\": rpc error: code = NotFound desc = could not find container \"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068\": container with ID starting with 40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068 not found: ID does not exist" Sep 16 12:07:43.334239 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.334194 2551 scope.go:117] "RemoveContainer" containerID="72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454" Sep 16 12:07:43.334485 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.334466 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454\": container with ID starting with 72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454 not found: ID does not exist" containerID="72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454" Sep 16 12:07:43.334707 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.334486 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454"} err="failed to get container status \"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454\": rpc error: code = NotFound desc = could not find container \"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454\": container with ID starting with 72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454 not found: ID does not exist" Sep 16 12:07:43.334707 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.334523 2551 scope.go:117] "RemoveContainer" containerID="f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358" Sep 16 12:07:43.334824 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:07:43.334763 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358\": container with ID starting with f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358 not found: ID does not exist" containerID="f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358" Sep 16 12:07:43.334824 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.334781 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358"} err="failed to get container status \"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358\": rpc error: code = NotFound desc = could not find container \"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358\": container with ID starting with f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358 not found: ID does not exist" Sep 16 12:07:43.334824 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.334796 2551 scope.go:117] "RemoveContainer" containerID="63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8" Sep 16 12:07:43.335041 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335020 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8"} err="failed to get container status \"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8\": rpc error: code = NotFound desc = could not find container \"63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8\": container with ID starting with 63d2b9398560a9df923c1e07fcd7dda22b250932898de4007b03b3cf6914bec8 not found: ID does not exist" Sep 16 12:07:43.335096 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335043 2551 scope.go:117] "RemoveContainer" containerID="026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40" Sep 16 12:07:43.335287 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335268 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40"} err="failed to get container status \"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40\": rpc error: code = NotFound desc = could not find container \"026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40\": container with ID starting with 026aab0de351d9622bc4f5f518616aac37e9b9be5fad032ced943a7e79163e40 not found: ID does not exist" Sep 16 12:07:43.335344 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335287 2551 scope.go:117] "RemoveContainer" containerID="efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525" Sep 16 12:07:43.335562 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335544 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525"} err="failed to get container status \"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525\": rpc error: code = NotFound desc = could not find container \"efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525\": container with ID starting with efe56a6c8c9be34d3c61f091211f736cf56a06d8f2cd05896443f36b95bbe525 not found: ID does not exist" Sep 16 12:07:43.335603 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335563 2551 scope.go:117] "RemoveContainer" containerID="40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068" Sep 16 12:07:43.335799 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335782 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068"} err="failed to get container status \"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068\": rpc error: code = NotFound desc = could not find container \"40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068\": container with ID starting with 40c62d0aeeee0e11adf5013c3d71ab57628e0e9dde147db85d4066afffb43068 not found: ID does not exist" Sep 16 12:07:43.335852 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.335799 2551 scope.go:117] "RemoveContainer" containerID="72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454" Sep 16 12:07:43.336027 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.336010 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454"} err="failed to get container status \"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454\": rpc error: code = NotFound desc = could not find container \"72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454\": container with ID starting with 72cd5df21cc7d7c673e4ab4e3a84a5821577f20fe57106b205c031ff0517d454 not found: ID does not exist" Sep 16 12:07:43.336070 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.336027 2551 scope.go:117] "RemoveContainer" containerID="f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358" Sep 16 12:07:43.336239 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.336223 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358"} err="failed to get container status \"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358\": rpc error: code = NotFound desc = could not find container \"f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358\": container with ID starting with f4a4c7ab452cfaab096e95e92a2ceede7b8859f6925a7fb8c08237bcd7af1358 not found: ID does not exist" Sep 16 12:07:43.389158 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389116 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-ui-tls\") pod \"10440004-48fd-49ac-b68b-6f9f202eecd1\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " Sep 16 12:07:43.389158 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389157 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls\") pod \"10440004-48fd-49ac-b68b-6f9f202eecd1\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " Sep 16 12:07:43.389407 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389189 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5sgm\" (UniqueName: \"kubernetes.io/projected/10440004-48fd-49ac-b68b-6f9f202eecd1-kube-api-access-p5sgm\") pod \"10440004-48fd-49ac-b68b-6f9f202eecd1\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " Sep 16 12:07:43.389407 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389212 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/10440004-48fd-49ac-b68b-6f9f202eecd1-feast-data\") pod \"10440004-48fd-49ac-b68b-6f9f202eecd1\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " Sep 16 12:07:43.389407 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389241 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls\") pod \"10440004-48fd-49ac-b68b-6f9f202eecd1\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " Sep 16 12:07:43.389407 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389268 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-offline-tls\") pod \"10440004-48fd-49ac-b68b-6f9f202eecd1\" (UID: \"10440004-48fd-49ac-b68b-6f9f202eecd1\") " Sep 16 12:07:43.389653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389523 2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-online-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.389653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389545 2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-feast-data\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.389653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389559 2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-ui-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.389653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389574 2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-offline-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.389653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389589 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9q4\" (UniqueName: \"kubernetes.io/projected/24b03bec-e7d0-4da5-940a-e2ba5ee0235a-kube-api-access-rx9q4\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.389956 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.389876 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10440004-48fd-49ac-b68b-6f9f202eecd1-feast-data" (OuterVolumeSpecName: "feast-data") pod "10440004-48fd-49ac-b68b-6f9f202eecd1" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:07:43.391470 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.391437 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "10440004-48fd-49ac-b68b-6f9f202eecd1" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.391860 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.391836 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "10440004-48fd-49ac-b68b-6f9f202eecd1" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.391860 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.391846 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10440004-48fd-49ac-b68b-6f9f202eecd1-kube-api-access-p5sgm" (OuterVolumeSpecName: "kube-api-access-p5sgm") pod "10440004-48fd-49ac-b68b-6f9f202eecd1" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1"). InnerVolumeSpecName "kube-api-access-p5sgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:07:43.391973 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.391855 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls" (OuterVolumeSpecName: "online-tls") pod "10440004-48fd-49ac-b68b-6f9f202eecd1" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.391973 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.391940 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "10440004-48fd-49ac-b68b-6f9f202eecd1" (UID: "10440004-48fd-49ac-b68b-6f9f202eecd1"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:07:43.490632 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.490544 2551 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-ui-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.490632 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.490577 2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-registry-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.490632 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.490590 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sgm\" (UniqueName: \"kubernetes.io/projected/10440004-48fd-49ac-b68b-6f9f202eecd1-kube-api-access-p5sgm\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.490632 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.490599 2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/10440004-48fd-49ac-b68b-6f9f202eecd1-feast-data\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.490632 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.490609 2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-online-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.490632 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.490617 2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/10440004-48fd-49ac-b68b-6f9f202eecd1-offline-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:07:43.561338 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.561304 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv"] Sep 16 12:07:43.567692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.567661 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-b469885bc-qkrrv"] Sep 16 12:07:43.579509 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.579464 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8"] Sep 16 12:07:43.583669 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:43.583643 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6d4b9548b7-rr5t8"] Sep 16 12:07:44.716903 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:44.716868 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" path="/var/lib/kubelet/pods/10440004-48fd-49ac-b68b-6f9f202eecd1/volumes" Sep 16 12:07:44.717518 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:44.717488 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" path="/var/lib/kubelet/pods/24b03bec-e7d0-4da5-940a-e2ba5ee0235a/volumes" Sep 16 12:07:55.589716 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.589681 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-njfws"] Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590004 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="feast-init" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590015 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="feast-init" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590027 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="feast-apply" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590032 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="feast-apply" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590041 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590046 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590053 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590058 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590065 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590070 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590082 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="feast-init" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590088 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="feast-init" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590094 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590099 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590107 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590112 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590118 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590123 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590128 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="offline" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590133 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="offline" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590140 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="feast-apply" Sep 16 12:07:55.590143 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590145 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="feast-apply" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590191 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="offline" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590200 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="ui" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590206 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="online" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590212 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="offline" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590219 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="registry" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590225 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="10440004-48fd-49ac-b68b-6f9f202eecd1" containerName="online" Sep 16 12:07:55.590879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.590231 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b03bec-e7d0-4da5-940a-e2ba5ee0235a" containerName="ui" Sep 16 12:07:55.595490 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.595470 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.597524 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.597479 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 16 12:07:55.597688 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.597638 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 16 12:07:55.597688 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.597639 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-sdr4n\"" Sep 16 12:07:55.598156 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.598135 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 16 12:07:55.601321 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.601292 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-njfws"] Sep 16 12:07:55.687445 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.687412 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/57e5809f-5c28-45c6-8c9b-f8529c2861d8-postgresdata\") pod \"postgres-864846d4f5-njfws\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.687639 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.687469 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86xv\" (UniqueName: \"kubernetes.io/projected/57e5809f-5c28-45c6-8c9b-f8529c2861d8-kube-api-access-w86xv\") pod \"postgres-864846d4f5-njfws\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.788748 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.788710 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/57e5809f-5c28-45c6-8c9b-f8529c2861d8-postgresdata\") pod \"postgres-864846d4f5-njfws\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.788905 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.788773 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86xv\" (UniqueName: \"kubernetes.io/projected/57e5809f-5c28-45c6-8c9b-f8529c2861d8-kube-api-access-w86xv\") pod \"postgres-864846d4f5-njfws\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.789119 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.789098 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/57e5809f-5c28-45c6-8c9b-f8529c2861d8-postgresdata\") pod \"postgres-864846d4f5-njfws\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.796935 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.796903 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86xv\" (UniqueName: \"kubernetes.io/projected/57e5809f-5c28-45c6-8c9b-f8529c2861d8-kube-api-access-w86xv\") pod \"postgres-864846d4f5-njfws\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:55.912666 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:55.912633 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:07:56.181711 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:56.181582 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-njfws"] Sep 16 12:07:56.184320 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:07:56.184290 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57e5809f_5c28_45c6_8c9b_f8529c2861d8.slice/crio-204508e70ee27a55b1893f4fe2715659839de22e02ba652abfcf004d1dcee9d4 WatchSource:0}: Error finding container 204508e70ee27a55b1893f4fe2715659839de22e02ba652abfcf004d1dcee9d4: Status 404 returned error can't find the container with id 204508e70ee27a55b1893f4fe2715659839de22e02ba652abfcf004d1dcee9d4 Sep 16 12:07:56.186195 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:56.186176 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 12:07:56.296359 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:07:56.296325 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-njfws" event={"ID":"57e5809f-5c28-45c6-8c9b-f8529c2861d8","Type":"ContainerStarted","Data":"204508e70ee27a55b1893f4fe2715659839de22e02ba652abfcf004d1dcee9d4"} Sep 16 12:08:02.325015 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:02.324961 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-njfws" event={"ID":"57e5809f-5c28-45c6-8c9b-f8529c2861d8","Type":"ContainerStarted","Data":"9e3dc0770567c0a1447e33e90ec5186afa6fdf5e2f89bab1edaa3e661216398d"} Sep 16 12:08:02.342812 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:02.342743 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-njfws" podStartSLOduration=1.821937101 podStartE2EDuration="7.342717755s" totalImagesPullingTime="5.520780654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:02.340514243 +0000 UTC m=+768.125787808" watchObservedRunningTime="2026-09-16 12:08:02.342717755 +0000 UTC m=+768.127991325" Sep 16 12:08:06.588585 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.588545 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt"] Sep 16 12:08:06.594380 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.594358 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.596723 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.596682 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 16 12:08:06.596868 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.596690 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 16 12:08:06.596868 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.596809 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-jhkn8\"" Sep 16 12:08:06.596868 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.596841 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 16 12:08:06.597020 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.596959 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 16 12:08:06.604574 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.604551 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt"] Sep 16 12:08:06.697751 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.697714 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-registry-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.697751 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.697750 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3d1e83-95f5-4e5e-927a-8234cad95528-feast-data\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.697949 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.697792 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdrt\" (UniqueName: \"kubernetes.io/projected/6e3d1e83-95f5-4e5e-927a-8234cad95528-kube-api-access-qcdrt\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.697949 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.697823 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.697949 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.697850 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-offline-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.798716 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.798684 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdrt\" (UniqueName: \"kubernetes.io/projected/6e3d1e83-95f5-4e5e-927a-8234cad95528-kube-api-access-qcdrt\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.798884 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.798738 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.798884 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.798778 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-offline-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.798884 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.798819 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-registry-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.798884 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.798863 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3d1e83-95f5-4e5e-927a-8234cad95528-feast-data\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.799016 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:08:06.798890 2551 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 16 12:08:06.799016 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:08:06.798961 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls podName:6e3d1e83-95f5-4e5e-927a-8234cad95528 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:07.298942071 +0000 UTC m=+773.084215620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls") pod "feast-credit-scoring-9dd5c994-5kdzt" (UID: "6e3d1e83-95f5-4e5e-927a-8234cad95528") : secret "feast-credit-scoring-online-tls" not found Sep 16 12:08:06.799311 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.799289 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3d1e83-95f5-4e5e-927a-8234cad95528-feast-data\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.801203 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.801176 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-registry-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.801324 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.801291 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-offline-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:06.808805 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:06.808782 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdrt\" (UniqueName: \"kubernetes.io/projected/6e3d1e83-95f5-4e5e-927a-8234cad95528-kube-api-access-qcdrt\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:07.304184 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:07.304143 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:07.306640 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:07.306620 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls\") pod \"feast-credit-scoring-9dd5c994-5kdzt\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:07.505124 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:07.505076 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:07.785697 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:07.785669 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt"] Sep 16 12:08:07.787593 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:08:07.787563 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3d1e83_95f5_4e5e_927a_8234cad95528.slice/crio-1b8b0458e3507db6c1b714f0f7194c6c4b9988386451becc3882e9d1ebbd609e WatchSource:0}: Error finding container 1b8b0458e3507db6c1b714f0f7194c6c4b9988386451becc3882e9d1ebbd609e: Status 404 returned error can't find the container with id 1b8b0458e3507db6c1b714f0f7194c6c4b9988386451becc3882e9d1ebbd609e Sep 16 12:08:08.354105 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:08.354064 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerStarted","Data":"94607bba69aec0c34d1500f791c5473ec9f5dc77486e03d4470bcea5bece7562"} Sep 16 12:08:08.354307 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:08.354156 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerStarted","Data":"1b8b0458e3507db6c1b714f0f7194c6c4b9988386451becc3882e9d1ebbd609e"} Sep 16 12:08:09.359733 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:09.359699 2551 generic.go:356] "Generic (PLEG): container finished" podID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerID="94607bba69aec0c34d1500f791c5473ec9f5dc77486e03d4470bcea5bece7562" exitCode=0 Sep 16 12:08:09.360113 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:09.359783 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerDied","Data":"94607bba69aec0c34d1500f791c5473ec9f5dc77486e03d4470bcea5bece7562"} Sep 16 12:08:10.366276 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:10.366240 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerStarted","Data":"a2b86cc5cf7c0013edb32a63a47d849611233c0bfd46497129962777ee05c563"} Sep 16 12:08:17.397369 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:17.397332 2551 generic.go:356] "Generic (PLEG): container finished" podID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerID="a2b86cc5cf7c0013edb32a63a47d849611233c0bfd46497129962777ee05c563" exitCode=0 Sep 16 12:08:17.397798 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:17.397404 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerDied","Data":"a2b86cc5cf7c0013edb32a63a47d849611233c0bfd46497129962777ee05c563"} Sep 16 12:08:17.401693 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:17.401669 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 16 12:08:18.407843 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:18.407797 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerStarted","Data":"45c0bf879d4ece6ff02c5a3fb3355a46cacecf94e59be4b489f62618945c3297"} Sep 16 12:08:18.408336 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:18.407882 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerStarted","Data":"bb0ecdeb921ffd223f23ecfe9d399ec9fabe7a72fd6a9d9f46a74aeb38c6b80f"} Sep 16 12:08:18.408336 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:18.407914 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerStarted","Data":"506181845e4e68261d7489c68cd691698e91e5b3910550eee1d571e76cbc7159"} Sep 16 12:08:18.429646 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:18.429572 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podStartSLOduration=12.429547896 podStartE2EDuration="12.429547896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:18.428977638 +0000 UTC m=+784.214251189" watchObservedRunningTime="2026-09-16 12:08:18.429547896 +0000 UTC m=+784.214821467" Sep 16 12:08:19.505288 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:19.505244 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:19.505793 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:19.505329 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:19.505793 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:19.505355 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:19.507048 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:19.507013 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 16 12:08:19.507168 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:19.507056 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 16 12:08:19.507168 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:19.507047 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 16 12:08:22.506260 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:22.506203 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 16 12:08:22.506260 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:22.506257 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 16 12:08:22.506823 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:22.506203 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 16 12:08:25.505574 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:25.505514 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 16 12:08:25.506080 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:25.505541 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.25:8816: connect: connection refused" Sep 16 12:08:25.506080 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:25.505539 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 16 12:08:27.506197 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:27.506154 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:27.506762 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:27.506241 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:27.506762 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:27.506265 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:28.505863 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:28.505798 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 16 12:08:28.506899 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:28.506861 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:28.508092 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:28.508057 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:28.514686 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:28.514652 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:29.479360 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:29.479329 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:31.506199 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:31.506149 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.25:6571: connect: connection refused" Sep 16 12:08:34.506776 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:34.506745 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:08:34.507553 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:08:34.507534 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:09:44.637355 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.637315 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt"] Sep 16 12:09:44.637894 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.637613 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" containerID="cri-o://45c0bf879d4ece6ff02c5a3fb3355a46cacecf94e59be4b489f62618945c3297" gracePeriod=30 Sep 16 12:09:44.637894 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.637625 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" containerID="cri-o://bb0ecdeb921ffd223f23ecfe9d399ec9fabe7a72fd6a9d9f46a74aeb38c6b80f" gracePeriod=30 Sep 16 12:09:44.637894 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.637602 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" containerID="cri-o://506181845e4e68261d7489c68cd691698e91e5b3910550eee1d571e76cbc7159" gracePeriod=30 Sep 16 12:09:44.652577 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.652539 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-njfws"] Sep 16 12:09:44.652805 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.652761 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-njfws" podUID="57e5809f-5c28-45c6-8c9b-f8529c2861d8" containerName="postgres" containerID="cri-o://9e3dc0770567c0a1447e33e90ec5186afa6fdf5e2f89bab1edaa3e661216398d" gracePeriod=30 Sep 16 12:09:44.813125 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.813088 2551 generic.go:356] "Generic (PLEG): container finished" podID="57e5809f-5c28-45c6-8c9b-f8529c2861d8" containerID="9e3dc0770567c0a1447e33e90ec5186afa6fdf5e2f89bab1edaa3e661216398d" exitCode=0 Sep 16 12:09:44.813277 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.813147 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-njfws" event={"ID":"57e5809f-5c28-45c6-8c9b-f8529c2861d8","Type":"ContainerDied","Data":"9e3dc0770567c0a1447e33e90ec5186afa6fdf5e2f89bab1edaa3e661216398d"} Sep 16 12:09:44.905765 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.905731 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:09:44.995545 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.995507 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w86xv\" (UniqueName: \"kubernetes.io/projected/57e5809f-5c28-45c6-8c9b-f8529c2861d8-kube-api-access-w86xv\") pod \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " Sep 16 12:09:44.995722 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.995585 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/57e5809f-5c28-45c6-8c9b-f8529c2861d8-postgresdata\") pod \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\" (UID: \"57e5809f-5c28-45c6-8c9b-f8529c2861d8\") " Sep 16 12:09:44.997667 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:44.997634 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e5809f-5c28-45c6-8c9b-f8529c2861d8-kube-api-access-w86xv" (OuterVolumeSpecName: "kube-api-access-w86xv") pod "57e5809f-5c28-45c6-8c9b-f8529c2861d8" (UID: "57e5809f-5c28-45c6-8c9b-f8529c2861d8"). InnerVolumeSpecName "kube-api-access-w86xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:09:45.031009 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.030960 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e5809f-5c28-45c6-8c9b-f8529c2861d8-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "57e5809f-5c28-45c6-8c9b-f8529c2861d8" (UID: "57e5809f-5c28-45c6-8c9b-f8529c2861d8"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:09:45.096332 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.096279 2551 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/57e5809f-5c28-45c6-8c9b-f8529c2861d8-postgresdata\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:09:45.096332 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.096327 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86xv\" (UniqueName: \"kubernetes.io/projected/57e5809f-5c28-45c6-8c9b-f8529c2861d8-kube-api-access-w86xv\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:09:45.820284 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.820243 2551 generic.go:356] "Generic (PLEG): container finished" podID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerID="bb0ecdeb921ffd223f23ecfe9d399ec9fabe7a72fd6a9d9f46a74aeb38c6b80f" exitCode=0 Sep 16 12:09:45.820755 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.820310 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerDied","Data":"bb0ecdeb921ffd223f23ecfe9d399ec9fabe7a72fd6a9d9f46a74aeb38c6b80f"} Sep 16 12:09:45.821648 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.821625 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-njfws" event={"ID":"57e5809f-5c28-45c6-8c9b-f8529c2861d8","Type":"ContainerDied","Data":"204508e70ee27a55b1893f4fe2715659839de22e02ba652abfcf004d1dcee9d4"} Sep 16 12:09:45.821762 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.821649 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-njfws" Sep 16 12:09:45.821762 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.821662 2551 scope.go:117] "RemoveContainer" containerID="9e3dc0770567c0a1447e33e90ec5186afa6fdf5e2f89bab1edaa3e661216398d" Sep 16 12:09:45.840714 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.840686 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-njfws"] Sep 16 12:09:45.843894 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:45.843868 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-njfws"] Sep 16 12:09:46.716757 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:46.716711 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e5809f-5c28-45c6-8c9b-f8529c2861d8" path="/var/lib/kubelet/pods/57e5809f-5c28-45c6-8c9b-f8529c2861d8/volumes" Sep 16 12:09:49.476136 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:49.476089 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 16 12:09:59.475698 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:09:59.475650 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 16 12:10:09.475656 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:09.475558 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" probeResult="failure" output="Get \"https://10.134.0.25:6567/health\": dial tcp 10.134.0.25:6567: connect: connection refused" Sep 16 12:10:09.476050 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:09.475702 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:10:14.949343 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:14.949238 2551 generic.go:356] "Generic (PLEG): container finished" podID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerID="45c0bf879d4ece6ff02c5a3fb3355a46cacecf94e59be4b489f62618945c3297" exitCode=137 Sep 16 12:10:14.949343 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:14.949280 2551 generic.go:356] "Generic (PLEG): container finished" podID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerID="506181845e4e68261d7489c68cd691698e91e5b3910550eee1d571e76cbc7159" exitCode=137 Sep 16 12:10:14.949343 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:14.949317 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerDied","Data":"45c0bf879d4ece6ff02c5a3fb3355a46cacecf94e59be4b489f62618945c3297"} Sep 16 12:10:14.949791 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:14.949383 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerDied","Data":"506181845e4e68261d7489c68cd691698e91e5b3910550eee1d571e76cbc7159"} Sep 16 12:10:15.292336 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.292305 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:10:15.349281 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.349243 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-registry-tls\") pod \"6e3d1e83-95f5-4e5e-927a-8234cad95528\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " Sep 16 12:10:15.349463 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.349330 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3d1e83-95f5-4e5e-927a-8234cad95528-feast-data\") pod \"6e3d1e83-95f5-4e5e-927a-8234cad95528\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " Sep 16 12:10:15.349463 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.349355 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-offline-tls\") pod \"6e3d1e83-95f5-4e5e-927a-8234cad95528\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " Sep 16 12:10:15.349463 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.349384 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls\") pod \"6e3d1e83-95f5-4e5e-927a-8234cad95528\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " Sep 16 12:10:15.349463 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.349444 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qcdrt\" (UniqueName: \"kubernetes.io/projected/6e3d1e83-95f5-4e5e-927a-8234cad95528-kube-api-access-qcdrt\") pod \"6e3d1e83-95f5-4e5e-927a-8234cad95528\" (UID: \"6e3d1e83-95f5-4e5e-927a-8234cad95528\") " Sep 16 12:10:15.351623 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.351585 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "6e3d1e83-95f5-4e5e-927a-8234cad95528" (UID: "6e3d1e83-95f5-4e5e-927a-8234cad95528"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:10:15.351769 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.351633 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls" (OuterVolumeSpecName: "online-tls") pod "6e3d1e83-95f5-4e5e-927a-8234cad95528" (UID: "6e3d1e83-95f5-4e5e-927a-8234cad95528"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:10:15.351769 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.351710 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3d1e83-95f5-4e5e-927a-8234cad95528-kube-api-access-qcdrt" (OuterVolumeSpecName: "kube-api-access-qcdrt") pod "6e3d1e83-95f5-4e5e-927a-8234cad95528" (UID: "6e3d1e83-95f5-4e5e-927a-8234cad95528"). InnerVolumeSpecName "kube-api-access-qcdrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:10:15.351864 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.351770 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6e3d1e83-95f5-4e5e-927a-8234cad95528" (UID: "6e3d1e83-95f5-4e5e-927a-8234cad95528"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:10:15.357835 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.357806 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3d1e83-95f5-4e5e-927a-8234cad95528-feast-data" (OuterVolumeSpecName: "feast-data") pod "6e3d1e83-95f5-4e5e-927a-8234cad95528" (UID: "6e3d1e83-95f5-4e5e-927a-8234cad95528"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:10:15.450401 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.450363 2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-registry-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:10:15.450401 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.450394 2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3d1e83-95f5-4e5e-927a-8234cad95528-feast-data\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:10:15.450401 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.450405 2551 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-offline-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:10:15.450401 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.450415 2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/6e3d1e83-95f5-4e5e-927a-8234cad95528-online-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:10:15.450707 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.450424 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdrt\" (UniqueName: \"kubernetes.io/projected/6e3d1e83-95f5-4e5e-927a-8234cad95528-kube-api-access-qcdrt\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:10:15.956348 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.956315 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" Sep 16 12:10:15.956348 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.956327 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt" event={"ID":"6e3d1e83-95f5-4e5e-927a-8234cad95528","Type":"ContainerDied","Data":"1b8b0458e3507db6c1b714f0f7194c6c4b9988386451becc3882e9d1ebbd609e"} Sep 16 12:10:15.956904 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.956391 2551 scope.go:117] "RemoveContainer" containerID="45c0bf879d4ece6ff02c5a3fb3355a46cacecf94e59be4b489f62618945c3297" Sep 16 12:10:15.965464 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.965438 2551 scope.go:117] "RemoveContainer" containerID="bb0ecdeb921ffd223f23ecfe9d399ec9fabe7a72fd6a9d9f46a74aeb38c6b80f" Sep 16 12:10:15.973307 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.973276 2551 scope.go:117] "RemoveContainer" containerID="506181845e4e68261d7489c68cd691698e91e5b3910550eee1d571e76cbc7159" Sep 16 12:10:15.980792 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.980768 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt"] Sep 16 12:10:15.981555 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.981537 2551 scope.go:117] "RemoveContainer" containerID="a2b86cc5cf7c0013edb32a63a47d849611233c0bfd46497129962777ee05c563" Sep 16 12:10:15.986675 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.986653 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-9dd5c994-5kdzt"] Sep 16 12:10:15.989603 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:15.989584 2551 scope.go:117] "RemoveContainer" containerID="94607bba69aec0c34d1500f791c5473ec9f5dc77486e03d4470bcea5bece7562" Sep 16 12:10:16.715515 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:10:16.715457 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" path="/var/lib/kubelet/pods/6e3d1e83-95f5-4e5e-927a-8234cad95528/volumes" Sep 16 12:13:18.828668 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.828629 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-hhvr6"] Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829134 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="feast-apply" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829151 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="feast-apply" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829168 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829174 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829188 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829193 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829203 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e5809f-5c28-45c6-8c9b-f8529c2861d8" containerName="postgres" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829209 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e5809f-5c28-45c6-8c9b-f8529c2861d8" containerName="postgres" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829217 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="feast-init" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829222 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="feast-init" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829228 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" Sep 16 12:13:18.829234 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829232 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" Sep 16 12:13:18.829635 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829284 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="online" Sep 16 12:13:18.829635 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829293 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="offline" Sep 16 12:13:18.829635 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829299 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e3d1e83-95f5-4e5e-927a-8234cad95528" containerName="registry" Sep 16 12:13:18.829635 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.829308 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e5809f-5c28-45c6-8c9b-f8529c2861d8" containerName="postgres" Sep 16 12:13:18.832041 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.832026 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:13:18.833971 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.833946 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 16 12:13:18.834090 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.833946 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 16 12:13:18.834423 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.834409 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-c9vnv\"" Sep 16 12:13:18.845811 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.845786 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-hhvr6"] Sep 16 12:13:18.899328 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:18.899291 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnfj\" (UniqueName: \"kubernetes.io/projected/a61e8a07-0211-42d9-80fd-fc5bf319bb98-kube-api-access-ncnfj\") pod \"redis-69bbd798b6-hhvr6\" (UID: \"a61e8a07-0211-42d9-80fd-fc5bf319bb98\") " pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:13:19.000664 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:19.000628 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnfj\" (UniqueName: \"kubernetes.io/projected/a61e8a07-0211-42d9-80fd-fc5bf319bb98-kube-api-access-ncnfj\") pod \"redis-69bbd798b6-hhvr6\" (UID: \"a61e8a07-0211-42d9-80fd-fc5bf319bb98\") " pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:13:19.008909 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:19.008873 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnfj\" (UniqueName: \"kubernetes.io/projected/a61e8a07-0211-42d9-80fd-fc5bf319bb98-kube-api-access-ncnfj\") pod \"redis-69bbd798b6-hhvr6\" (UID: \"a61e8a07-0211-42d9-80fd-fc5bf319bb98\") " pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:13:19.151975 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:19.151943 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:13:19.432542 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:19.432418 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-hhvr6"] Sep 16 12:13:19.435173 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:13:19.435142 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda61e8a07_0211_42d9_80fd_fc5bf319bb98.slice/crio-5d601a8c6e35387a272b4e1c49ebd72b4acc8ba46082f1b49bb78c2679a5edff WatchSource:0}: Error finding container 5d601a8c6e35387a272b4e1c49ebd72b4acc8ba46082f1b49bb78c2679a5edff: Status 404 returned error can't find the container with id 5d601a8c6e35387a272b4e1c49ebd72b4acc8ba46082f1b49bb78c2679a5edff Sep 16 12:13:19.437087 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:19.437069 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 12:13:19.711692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:19.711601 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" event={"ID":"a61e8a07-0211-42d9-80fd-fc5bf319bb98","Type":"ContainerStarted","Data":"5d601a8c6e35387a272b4e1c49ebd72b4acc8ba46082f1b49bb78c2679a5edff"} Sep 16 12:13:21.722328 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:21.722288 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" event={"ID":"a61e8a07-0211-42d9-80fd-fc5bf319bb98","Type":"ContainerStarted","Data":"7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73"} Sep 16 12:13:21.738010 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:21.737963 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" podStartSLOduration=2.046094966 podStartE2EDuration="3.737946554s" totalImagesPullingTime="1.691851588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:13:21.736540902 +0000 UTC m=+1087.521814472" watchObservedRunningTime="2026-09-16 12:13:21.737946554 +0000 UTC m=+1087.523220123" Sep 16 12:13:24.361555 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.361516 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrwbq"] Sep 16 12:13:24.364832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.364811 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.366816 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.366794 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 16 12:13:24.370446 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.370419 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrwbq"] Sep 16 12:13:24.446784 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.446744 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0e009238-d083-4bda-9e2e-3c75dd76f272-postgresdata\") pod \"postgres-864846d4f5-nrwbq\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.446968 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.446796 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9l5t\" (UniqueName: \"kubernetes.io/projected/0e009238-d083-4bda-9e2e-3c75dd76f272-kube-api-access-h9l5t\") pod \"postgres-864846d4f5-nrwbq\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.547318 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.547294 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0e009238-d083-4bda-9e2e-3c75dd76f272-postgresdata\") pod \"postgres-864846d4f5-nrwbq\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.547426 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.547334 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9l5t\" (UniqueName: \"kubernetes.io/projected/0e009238-d083-4bda-9e2e-3c75dd76f272-kube-api-access-h9l5t\") pod \"postgres-864846d4f5-nrwbq\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.547718 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.547698 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0e009238-d083-4bda-9e2e-3c75dd76f272-postgresdata\") pod \"postgres-864846d4f5-nrwbq\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.556448 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.556409 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9l5t\" (UniqueName: \"kubernetes.io/projected/0e009238-d083-4bda-9e2e-3c75dd76f272-kube-api-access-h9l5t\") pod \"postgres-864846d4f5-nrwbq\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.675281 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.675241 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:13:24.942640 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:24.942613 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrwbq"] Sep 16 12:13:24.945201 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:13:24.945166 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e009238_d083_4bda_9e2e_3c75dd76f272.slice/crio-0c4452d51520851ad08ad00504116f939ac7d9147c98728e42b1e7a896b9b238 WatchSource:0}: Error finding container 0c4452d51520851ad08ad00504116f939ac7d9147c98728e42b1e7a896b9b238: Status 404 returned error can't find the container with id 0c4452d51520851ad08ad00504116f939ac7d9147c98728e42b1e7a896b9b238 Sep 16 12:13:25.741804 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:25.741769 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" event={"ID":"0e009238-d083-4bda-9e2e-3c75dd76f272","Type":"ContainerStarted","Data":"28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397"} Sep 16 12:13:25.741804 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:25.741810 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" event={"ID":"0e009238-d083-4bda-9e2e-3c75dd76f272","Type":"ContainerStarted","Data":"0c4452d51520851ad08ad00504116f939ac7d9147c98728e42b1e7a896b9b238"} Sep 16 12:13:25.756021 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:25.755962 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" podStartSLOduration=1.755941336 podStartE2EDuration="1.755941336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:13:25.754588848 +0000 UTC m=+1091.539862416" watchObservedRunningTime="2026-09-16 12:13:25.755941336 +0000 UTC m=+1091.541214904" Sep 16 12:13:30.040612 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.040579 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf"] Sep 16 12:13:30.044175 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.044154 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.048191 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.048169 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 16 12:13:30.048328 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.048167 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 16 12:13:30.048328 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.048169 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-dwvtt\"" Sep 16 12:13:30.048527 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.048492 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 16 12:13:30.057255 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.057231 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf"] Sep 16 12:13:30.201883 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.201836 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-registry-tls\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.202090 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.201911 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c18fbc-be27-4f86-ad76-6a40964d02f9-feast-data\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.202090 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.201944 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-online-tls\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.202090 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.201977 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lp7l\" (UniqueName: \"kubernetes.io/projected/b5c18fbc-be27-4f86-ad76-6a40964d02f9-kube-api-access-4lp7l\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.302983 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.302890 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-registry-tls\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.302983 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.302954 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c18fbc-be27-4f86-ad76-6a40964d02f9-feast-data\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.302983 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.302978 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-online-tls\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.303207 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.303004 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lp7l\" (UniqueName: \"kubernetes.io/projected/b5c18fbc-be27-4f86-ad76-6a40964d02f9-kube-api-access-4lp7l\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.303558 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.303536 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c18fbc-be27-4f86-ad76-6a40964d02f9-feast-data\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.305692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.305667 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-registry-tls\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.305824 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.305667 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-online-tls\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.310585 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.310563 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lp7l\" (UniqueName: \"kubernetes.io/projected/b5c18fbc-be27-4f86-ad76-6a40964d02f9-kube-api-access-4lp7l\") pod \"feast-credit-scoring-755cffc58f-m9rbf\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.353775 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.353738 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:30.626864 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.626654 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf"] Sep 16 12:13:30.629147 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:13:30.629113 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c18fbc_be27_4f86_ad76_6a40964d02f9.slice/crio-a5e1563a544f618aed30a2cee58559ca410efb3011dffb63a48b19df04a9ec54 WatchSource:0}: Error finding container a5e1563a544f618aed30a2cee58559ca410efb3011dffb63a48b19df04a9ec54: Status 404 returned error can't find the container with id a5e1563a544f618aed30a2cee58559ca410efb3011dffb63a48b19df04a9ec54 Sep 16 12:13:30.763192 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.763153 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerStarted","Data":"bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0"} Sep 16 12:13:30.763403 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:30.763206 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerStarted","Data":"a5e1563a544f618aed30a2cee58559ca410efb3011dffb63a48b19df04a9ec54"} Sep 16 12:13:32.777273 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:32.777233 2551 generic.go:356] "Generic (PLEG): container finished" podID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerID="bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0" exitCode=0 Sep 16 12:13:32.777668 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:32.777306 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerDied","Data":"bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0"} Sep 16 12:13:33.783855 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:33.783812 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerStarted","Data":"322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61"} Sep 16 12:13:40.816774 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:40.816735 2551 generic.go:356] "Generic (PLEG): container finished" podID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerID="322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61" exitCode=0 Sep 16 12:13:40.817288 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:40.816781 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerDied","Data":"322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61"} Sep 16 12:13:41.825715 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:41.825673 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerStarted","Data":"eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6"} Sep 16 12:13:41.826093 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:41.825752 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerStarted","Data":"86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12"} Sep 16 12:13:41.852129 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:41.852058 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podStartSLOduration=11.852036016 podStartE2EDuration="11.852036016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:13:41.851260117 +0000 UTC m=+1107.636533707" watchObservedRunningTime="2026-09-16 12:13:41.852036016 +0000 UTC m=+1107.637309585" Sep 16 12:13:42.354151 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:42.354112 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:42.354318 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:42.354196 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:42.355673 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:42.355640 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 16 12:13:42.355799 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:42.355687 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 16 12:13:45.354666 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:45.354614 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 16 12:13:45.355082 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:45.354674 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 16 12:13:48.354640 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:48.354594 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 16 12:13:48.360834 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:48.360806 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:48.361307 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:48.361286 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:48.365397 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:48.365377 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:50.354634 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:50.354590 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:51.355071 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:51.355025 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Sep 16 12:13:54.354635 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:54.354603 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:54.355431 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:54.355407 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:13:56.004231 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.004197 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn"] Sep 16 12:13:56.007106 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.007086 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.009147 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.009122 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 16 12:13:56.009289 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.009165 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 16 12:13:56.009289 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.009182 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-rsfjx\"" Sep 16 12:13:56.009289 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.009209 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 16 12:13:56.020967 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.020944 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn"] Sep 16 12:13:56.043732 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.043698 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.043732 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.043730 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvb4\" (UniqueName: \"kubernetes.io/projected/574ae380-dbb7-441b-af3a-e6cdd4f434f1-kube-api-access-wcvb4\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.043936 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.043802 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.043936 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.043863 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/574ae380-dbb7-441b-af3a-e6cdd4f434f1-feast-data\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.145209 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.145166 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.145209 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.145209 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvb4\" (UniqueName: \"kubernetes.io/projected/574ae380-dbb7-441b-af3a-e6cdd4f434f1-kube-api-access-wcvb4\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.145447 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.145229 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.145447 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.145263 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/574ae380-dbb7-441b-af3a-e6cdd4f434f1-feast-data\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.145447 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:13:56.145319 2551 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 16 12:13:56.145447 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:13:56.145334 2551 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-online-tls: secret "feast-driver-ranking-online-tls" not found Sep 16 12:13:56.145447 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:13:56.145397 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls podName:574ae380-dbb7-441b-af3a-e6cdd4f434f1 nodeName:}" failed. No retries permitted until 2026-09-16 12:13:56.645380887 +0000 UTC m=+1122.430654438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls") pod "feast-driver-ranking-74bf7fb59d-d6bfn" (UID: "574ae380-dbb7-441b-af3a-e6cdd4f434f1") : secret "feast-driver-ranking-registry-tls" not found Sep 16 12:13:56.145447 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:13:56.145412 2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls podName:574ae380-dbb7-441b-af3a-e6cdd4f434f1 nodeName:}" failed. No retries permitted until 2026-09-16 12:13:56.645406155 +0000 UTC m=+1122.430679701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls") pod "feast-driver-ranking-74bf7fb59d-d6bfn" (UID: "574ae380-dbb7-441b-af3a-e6cdd4f434f1") : secret "feast-driver-ranking-online-tls" not found Sep 16 12:13:56.145746 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.145636 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/574ae380-dbb7-441b-af3a-e6cdd4f434f1-feast-data\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.167138 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.167100 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvb4\" (UniqueName: \"kubernetes.io/projected/574ae380-dbb7-441b-af3a-e6cdd4f434f1-kube-api-access-wcvb4\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.650413 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.650369 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.650413 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.650422 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.652847 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.652818 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.653003 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.652976 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls\") pod \"feast-driver-ranking-74bf7fb59d-d6bfn\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:56.916678 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:56.916581 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:13:57.191254 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:57.191226 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn"] Sep 16 12:13:57.193260 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:13:57.193225 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574ae380_dbb7_441b_af3a_e6cdd4f434f1.slice/crio-9271bd09c7f66a18bb2cc714e3d142fe7677bf19be0dcb0c5504deda9252b70a WatchSource:0}: Error finding container 9271bd09c7f66a18bb2cc714e3d142fe7677bf19be0dcb0c5504deda9252b70a: Status 404 returned error can't find the container with id 9271bd09c7f66a18bb2cc714e3d142fe7677bf19be0dcb0c5504deda9252b70a Sep 16 12:13:57.904065 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:57.904032 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerStarted","Data":"bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02"} Sep 16 12:13:57.904242 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:13:57.904081 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerStarted","Data":"9271bd09c7f66a18bb2cc714e3d142fe7677bf19be0dcb0c5504deda9252b70a"} Sep 16 12:14:02.933252 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:02.933216 2551 generic.go:356] "Generic (PLEG): container finished" podID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerID="bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02" exitCode=0 Sep 16 12:14:02.933657 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:02.933293 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerDied","Data":"bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02"} Sep 16 12:14:03.939960 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:03.939917 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerStarted","Data":"b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15"} Sep 16 12:14:10.974413 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:10.974369 2551 generic.go:356] "Generic (PLEG): container finished" podID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerID="b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15" exitCode=0 Sep 16 12:14:10.974927 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:10.974444 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerDied","Data":"b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15"} Sep 16 12:14:11.982918 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:11.982878 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerStarted","Data":"f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834"} Sep 16 12:14:11.983268 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:11.982993 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerStarted","Data":"af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed"} Sep 16 12:14:12.006507 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:12.006433 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podStartSLOduration=17.006400932 podStartE2EDuration="17.006400932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:12.004808297 +0000 UTC m=+1137.790081881" watchObservedRunningTime="2026-09-16 12:14:12.006400932 +0000 UTC m=+1137.791674500" Sep 16 12:14:14.917199 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:14.917158 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:14.917673 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:14.917245 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:14.919043 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:14.919005 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 16 12:14:14.919043 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:14.919028 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 16 12:14:16.917590 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:16.917546 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:16.918098 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:16.917612 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:17.917432 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:17.917381 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 16 12:14:17.930221 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:17.930187 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:17.934070 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:17.934038 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:20.917869 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:20.917827 2551 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.29:6571: connect: connection refused" Sep 16 12:14:23.917817 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:23.917769 2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:14:23.918539 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:14:23.918518 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:15:47.878845 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:47.878811 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf"] Sep 16 12:15:47.879375 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:47.879047 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" containerID="cri-o://86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12" gracePeriod=30 Sep 16 12:15:47.879375 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:47.879125 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" containerID="cri-o://eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6" gracePeriod=30 Sep 16 12:15:47.888341 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:47.888312 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn"] Sep 16 12:15:47.888579 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:47.888538 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" containerID="cri-o://af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed" gracePeriod=30 Sep 16 12:15:47.888712 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:47.888559 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" containerID="cri-o://f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834" gracePeriod=30 Sep 16 12:15:48.361737 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:48.361683 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 16 12:15:49.430925 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.430888 2551 generic.go:356] "Generic (PLEG): container finished" podID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerID="eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6" exitCode=0 Sep 16 12:15:49.431397 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.430956 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerDied","Data":"eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6"} Sep 16 12:15:49.433344 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.433321 2551 generic.go:356] "Generic (PLEG): container finished" podID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerID="f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834" exitCode=0 Sep 16 12:15:49.433466 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.433393 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerDied","Data":"f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834"} Sep 16 12:15:49.690087 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.690003 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrwbq"] Sep 16 12:15:49.690233 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.690188 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" podUID="0e009238-d083-4bda-9e2e-3c75dd76f272" containerName="postgres" containerID="cri-o://28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397" gracePeriod=30 Sep 16 12:15:49.694354 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.694325 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-hhvr6"] Sep 16 12:15:49.694550 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.694492 2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" podUID="a61e8a07-0211-42d9-80fd-fc5bf319bb98" containerName="redis" containerID="cri-o://7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73" gracePeriod=30 Sep 16 12:15:49.960895 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.960870 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:15:49.963626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:49.963604 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:15:50.076900 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.076864 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncnfj\" (UniqueName: \"kubernetes.io/projected/a61e8a07-0211-42d9-80fd-fc5bf319bb98-kube-api-access-ncnfj\") pod \"a61e8a07-0211-42d9-80fd-fc5bf319bb98\" (UID: \"a61e8a07-0211-42d9-80fd-fc5bf319bb98\") " Sep 16 12:15:50.076900 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.076900 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9l5t\" (UniqueName: \"kubernetes.io/projected/0e009238-d083-4bda-9e2e-3c75dd76f272-kube-api-access-h9l5t\") pod \"0e009238-d083-4bda-9e2e-3c75dd76f272\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " Sep 16 12:15:50.077128 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.076966 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0e009238-d083-4bda-9e2e-3c75dd76f272-postgresdata\") pod \"0e009238-d083-4bda-9e2e-3c75dd76f272\" (UID: \"0e009238-d083-4bda-9e2e-3c75dd76f272\") " Sep 16 12:15:50.079227 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.079185 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61e8a07-0211-42d9-80fd-fc5bf319bb98-kube-api-access-ncnfj" (OuterVolumeSpecName: "kube-api-access-ncnfj") pod "a61e8a07-0211-42d9-80fd-fc5bf319bb98" (UID: "a61e8a07-0211-42d9-80fd-fc5bf319bb98"). InnerVolumeSpecName "kube-api-access-ncnfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:15:50.079227 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.079213 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e009238-d083-4bda-9e2e-3c75dd76f272-kube-api-access-h9l5t" (OuterVolumeSpecName: "kube-api-access-h9l5t") pod "0e009238-d083-4bda-9e2e-3c75dd76f272" (UID: "0e009238-d083-4bda-9e2e-3c75dd76f272"). InnerVolumeSpecName "kube-api-access-h9l5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:15:50.107420 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.107374 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e009238-d083-4bda-9e2e-3c75dd76f272-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "0e009238-d083-4bda-9e2e-3c75dd76f272" (UID: "0e009238-d083-4bda-9e2e-3c75dd76f272"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:15:50.177904 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.177869 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncnfj\" (UniqueName: \"kubernetes.io/projected/a61e8a07-0211-42d9-80fd-fc5bf319bb98-kube-api-access-ncnfj\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:15:50.177904 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.177901 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9l5t\" (UniqueName: \"kubernetes.io/projected/0e009238-d083-4bda-9e2e-3c75dd76f272-kube-api-access-h9l5t\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:15:50.178055 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.177913 2551 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/0e009238-d083-4bda-9e2e-3c75dd76f272-postgresdata\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:15:50.439272 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.439234 2551 generic.go:356] "Generic (PLEG): container finished" podID="a61e8a07-0211-42d9-80fd-fc5bf319bb98" containerID="7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73" exitCode=0 Sep 16 12:15:50.439272 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.439272 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" Sep 16 12:15:50.439797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.439326 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" event={"ID":"a61e8a07-0211-42d9-80fd-fc5bf319bb98","Type":"ContainerDied","Data":"7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73"} Sep 16 12:15:50.439797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.439363 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-hhvr6" event={"ID":"a61e8a07-0211-42d9-80fd-fc5bf319bb98","Type":"ContainerDied","Data":"5d601a8c6e35387a272b4e1c49ebd72b4acc8ba46082f1b49bb78c2679a5edff"} Sep 16 12:15:50.439797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.439383 2551 scope.go:117] "RemoveContainer" containerID="7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73" Sep 16 12:15:50.440660 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.440634 2551 generic.go:356] "Generic (PLEG): container finished" podID="0e009238-d083-4bda-9e2e-3c75dd76f272" containerID="28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397" exitCode=0 Sep 16 12:15:50.440797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.440660 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" event={"ID":"0e009238-d083-4bda-9e2e-3c75dd76f272","Type":"ContainerDied","Data":"28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397"} Sep 16 12:15:50.440797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.440679 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" Sep 16 12:15:50.440797 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.440688 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-nrwbq" event={"ID":"0e009238-d083-4bda-9e2e-3c75dd76f272","Type":"ContainerDied","Data":"0c4452d51520851ad08ad00504116f939ac7d9147c98728e42b1e7a896b9b238"} Sep 16 12:15:50.449223 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.449205 2551 scope.go:117] "RemoveContainer" containerID="7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73" Sep 16 12:15:50.449603 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:15:50.449576 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73\": container with ID starting with 7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73 not found: ID does not exist" containerID="7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73" Sep 16 12:15:50.449708 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.449611 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73"} err="failed to get container status \"7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73\": rpc error: code = NotFound desc = could not find container \"7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73\": container with ID starting with 7c128044e1f7ea3f1f5cadd198f53267964ecd5c129b9c7e939949cffe7dcd73 not found: ID does not exist" Sep 16 12:15:50.449708 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.449634 2551 scope.go:117] "RemoveContainer" containerID="28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397" Sep 16 12:15:50.457879 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.457861 2551 scope.go:117] "RemoveContainer" containerID="28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397" Sep 16 12:15:50.458243 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:15:50.458177 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397\": container with ID starting with 28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397 not found: ID does not exist" containerID="28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397" Sep 16 12:15:50.458243 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.458208 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397"} err="failed to get container status \"28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397\": rpc error: code = NotFound desc = could not find container \"28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397\": container with ID starting with 28519a3c7acd70c2ce4cc5cbb6df1409fd62299e42786ed31ec2381314d47397 not found: ID does not exist" Sep 16 12:15:50.459028 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.459009 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-hhvr6"] Sep 16 12:15:50.462086 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.462063 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-hhvr6"] Sep 16 12:15:50.475212 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.475177 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrwbq"] Sep 16 12:15:50.480526 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.480484 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-nrwbq"] Sep 16 12:15:50.716636 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.716555 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e009238-d083-4bda-9e2e-3c75dd76f272" path="/var/lib/kubelet/pods/0e009238-d083-4bda-9e2e-3c75dd76f272/volumes" Sep 16 12:15:50.716893 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:50.716880 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61e8a07-0211-42d9-80fd-fc5bf319bb98" path="/var/lib/kubelet/pods/a61e8a07-0211-42d9-80fd-fc5bf319bb98/volumes" Sep 16 12:15:57.931039 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:57.930991 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 16 12:15:58.361781 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:15:58.361690 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 16 12:16:07.931293 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:07.931192 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 16 12:16:08.362678 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:08.362580 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 16 12:16:08.362831 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:08.362710 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:16:17.931001 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:17.930954 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Sep 16 12:16:17.931469 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:17.931105 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:16:18.361931 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.361816 2551 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Sep 16 12:16:18.527231 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.527204 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:16:18.558178 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.558142 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:16:18.564988 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.564954 2551 generic.go:356] "Generic (PLEG): container finished" podID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerID="86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12" exitCode=137 Sep 16 12:16:18.565180 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.565021 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" Sep 16 12:16:18.565180 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.565032 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerDied","Data":"86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12"} Sep 16 12:16:18.565180 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.565097 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf" event={"ID":"b5c18fbc-be27-4f86-ad76-6a40964d02f9","Type":"ContainerDied","Data":"a5e1563a544f618aed30a2cee58559ca410efb3011dffb63a48b19df04a9ec54"} Sep 16 12:16:18.565180 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.565129 2551 scope.go:117] "RemoveContainer" containerID="eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6" Sep 16 12:16:18.567579 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.567550 2551 generic.go:356] "Generic (PLEG): container finished" podID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerID="af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed" exitCode=137 Sep 16 12:16:18.567692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.567598 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerDied","Data":"af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed"} Sep 16 12:16:18.567692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.567607 2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" Sep 16 12:16:18.567692 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.567641 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn" event={"ID":"574ae380-dbb7-441b-af3a-e6cdd4f434f1","Type":"ContainerDied","Data":"9271bd09c7f66a18bb2cc714e3d142fe7677bf19be0dcb0c5504deda9252b70a"} Sep 16 12:16:18.574194 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.574170 2551 scope.go:117] "RemoveContainer" containerID="86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12" Sep 16 12:16:18.584574 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.584545 2551 scope.go:117] "RemoveContainer" containerID="322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61" Sep 16 12:16:18.592852 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.592830 2551 scope.go:117] "RemoveContainer" containerID="bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0" Sep 16 12:16:18.600921 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.600900 2551 scope.go:117] "RemoveContainer" containerID="eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6" Sep 16 12:16:18.601274 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.601254 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6\": container with ID starting with eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6 not found: ID does not exist" containerID="eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6" Sep 16 12:16:18.601342 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.601286 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6"} err="failed to get container status \"eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6\": rpc error: code = NotFound desc = could not find container \"eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6\": container with ID starting with eaf67184f8ca06d2c1b540c2f3ea07072a7f66a429ae3c5f3104715c6290a8b6 not found: ID does not exist" Sep 16 12:16:18.601342 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.601305 2551 scope.go:117] "RemoveContainer" containerID="86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12" Sep 16 12:16:18.601606 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.601590 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12\": container with ID starting with 86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12 not found: ID does not exist" containerID="86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12" Sep 16 12:16:18.601653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.601619 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12"} err="failed to get container status \"86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12\": rpc error: code = NotFound desc = could not find container \"86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12\": container with ID starting with 86c2f07882b1c80fe0e70523d410a0ea6214a4628baf31acb319016b6c1d7a12 not found: ID does not exist" Sep 16 12:16:18.601653 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.601632 2551 scope.go:117] "RemoveContainer" containerID="322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61" Sep 16 12:16:18.601914 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.601897 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61\": container with ID starting with 322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61 not found: ID does not exist" containerID="322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61" Sep 16 12:16:18.601952 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.601920 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61"} err="failed to get container status \"322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61\": rpc error: code = NotFound desc = could not find container \"322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61\": container with ID starting with 322c2a1ac387fc91172d44da61e2e19ada3466649d5fc28a01ac3ceee1241b61 not found: ID does not exist" Sep 16 12:16:18.601952 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.601937 2551 scope.go:117] "RemoveContainer" containerID="bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0" Sep 16 12:16:18.602171 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.602157 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0\": container with ID starting with bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0 not found: ID does not exist" containerID="bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0" Sep 16 12:16:18.602210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.602174 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0"} err="failed to get container status \"bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0\": rpc error: code = NotFound desc = could not find container \"bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0\": container with ID starting with bdf6f09cc3e68854465421793295d835b2c65495fa26b5cb180f417218b92ba0 not found: ID does not exist" Sep 16 12:16:18.602210 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.602191 2551 scope.go:117] "RemoveContainer" containerID="f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834" Sep 16 12:16:18.610344 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.610326 2551 scope.go:117] "RemoveContainer" containerID="af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed" Sep 16 12:16:18.618803 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.618783 2551 scope.go:117] "RemoveContainer" containerID="b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15" Sep 16 12:16:18.623981 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.623959 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls\") pod \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " Sep 16 12:16:18.624093 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.623992 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-registry-tls\") pod \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " Sep 16 12:16:18.624093 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624019 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c18fbc-be27-4f86-ad76-6a40964d02f9-feast-data\") pod \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " Sep 16 12:16:18.624093 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624033 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/574ae380-dbb7-441b-af3a-e6cdd4f434f1-feast-data\") pod \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " Sep 16 12:16:18.624227 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624104 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvb4\" (UniqueName: \"kubernetes.io/projected/574ae380-dbb7-441b-af3a-e6cdd4f434f1-kube-api-access-wcvb4\") pod \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " Sep 16 12:16:18.624227 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624155 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lp7l\" (UniqueName: \"kubernetes.io/projected/b5c18fbc-be27-4f86-ad76-6a40964d02f9-kube-api-access-4lp7l\") pod \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " Sep 16 12:16:18.624227 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624204 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-online-tls\") pod \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\" (UID: \"b5c18fbc-be27-4f86-ad76-6a40964d02f9\") " Sep 16 12:16:18.624385 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624231 2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls\") pod \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\" (UID: \"574ae380-dbb7-441b-af3a-e6cdd4f434f1\") " Sep 16 12:16:18.624600 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.624569 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574ae380-dbb7-441b-af3a-e6cdd4f434f1-feast-data" (OuterVolumeSpecName: "feast-data") pod "574ae380-dbb7-441b-af3a-e6cdd4f434f1" (UID: "574ae380-dbb7-441b-af3a-e6cdd4f434f1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:16:18.626672 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.626638 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "574ae380-dbb7-441b-af3a-e6cdd4f434f1" (UID: "574ae380-dbb7-441b-af3a-e6cdd4f434f1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:16:18.626672 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.626662 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b5c18fbc-be27-4f86-ad76-6a40964d02f9" (UID: "b5c18fbc-be27-4f86-ad76-6a40964d02f9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:16:18.626849 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.626730 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574ae380-dbb7-441b-af3a-e6cdd4f434f1-kube-api-access-wcvb4" (OuterVolumeSpecName: "kube-api-access-wcvb4") pod "574ae380-dbb7-441b-af3a-e6cdd4f434f1" (UID: "574ae380-dbb7-441b-af3a-e6cdd4f434f1"). InnerVolumeSpecName "kube-api-access-wcvb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:16:18.626929 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.626906 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-online-tls" (OuterVolumeSpecName: "online-tls") pod "b5c18fbc-be27-4f86-ad76-6a40964d02f9" (UID: "b5c18fbc-be27-4f86-ad76-6a40964d02f9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:16:18.627036 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.627019 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c18fbc-be27-4f86-ad76-6a40964d02f9-kube-api-access-4lp7l" (OuterVolumeSpecName: "kube-api-access-4lp7l") pod "b5c18fbc-be27-4f86-ad76-6a40964d02f9" (UID: "b5c18fbc-be27-4f86-ad76-6a40964d02f9"). InnerVolumeSpecName "kube-api-access-4lp7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 12:16:18.627198 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.627184 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls" (OuterVolumeSpecName: "online-tls") pod "574ae380-dbb7-441b-af3a-e6cdd4f434f1" (UID: "574ae380-dbb7-441b-af3a-e6cdd4f434f1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 12:16:18.627862 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.627836 2551 scope.go:117] "RemoveContainer" containerID="bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02" Sep 16 12:16:18.634387 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.634345 2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5c18fbc-be27-4f86-ad76-6a40964d02f9-feast-data" (OuterVolumeSpecName: "feast-data") pod "b5c18fbc-be27-4f86-ad76-6a40964d02f9" (UID: "b5c18fbc-be27-4f86-ad76-6a40964d02f9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 12:16:18.635398 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.635382 2551 scope.go:117] "RemoveContainer" containerID="f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834" Sep 16 12:16:18.635716 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.635696 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834\": container with ID starting with f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834 not found: ID does not exist" containerID="f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834" Sep 16 12:16:18.635787 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.635729 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834"} err="failed to get container status \"f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834\": rpc error: code = NotFound desc = could not find container \"f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834\": container with ID starting with f1821a5fa1d23a5728758f67c5fd721bee28f250b5b3ec2b4e326e57a3a24834 not found: ID does not exist" Sep 16 12:16:18.635787 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.635749 2551 scope.go:117] "RemoveContainer" containerID="af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed" Sep 16 12:16:18.636030 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.636013 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed\": container with ID starting with af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed not found: ID does not exist" containerID="af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed" Sep 16 12:16:18.636094 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.636040 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed"} err="failed to get container status \"af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed\": rpc error: code = NotFound desc = could not find container \"af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed\": container with ID starting with af599448d63b5802e3263c9c3e855a0e485f2c3fa80e417f15b09dcb8c2ba7ed not found: ID does not exist" Sep 16 12:16:18.636094 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.636064 2551 scope.go:117] "RemoveContainer" containerID="b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15" Sep 16 12:16:18.636328 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.636308 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15\": container with ID starting with b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15 not found: ID does not exist" containerID="b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15" Sep 16 12:16:18.636368 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.636334 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15"} err="failed to get container status \"b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15\": rpc error: code = NotFound desc = could not find container \"b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15\": container with ID starting with b4ce7209b9cab0e324631dd88f985ccf69a9053d4214f3b13abd3463705cbd15 not found: ID does not exist" Sep 16 12:16:18.636368 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.636346 2551 scope.go:117] "RemoveContainer" containerID="bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02" Sep 16 12:16:18.636676 ip-10-0-140-32 kubenswrapper[2551]: E0916 12:16:18.636653 2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02\": container with ID starting with bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02 not found: ID does not exist" containerID="bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02" Sep 16 12:16:18.636751 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.636684 2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02"} err="failed to get container status \"bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02\": rpc error: code = NotFound desc = could not find container \"bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02\": container with ID starting with bdb0527c9db4d11d916f366d620cad73464a37fec042377756bbf98469ce2a02 not found: ID does not exist" Sep 16 12:16:18.724983 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.724955 2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-registry-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.724983 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.724982 2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-registry-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.725205 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.724993 2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b5c18fbc-be27-4f86-ad76-6a40964d02f9-feast-data\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.725205 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.725005 2551 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/574ae380-dbb7-441b-af3a-e6cdd4f434f1-feast-data\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.725205 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.725017 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvb4\" (UniqueName: \"kubernetes.io/projected/574ae380-dbb7-441b-af3a-e6cdd4f434f1-kube-api-access-wcvb4\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.725205 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.725030 2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lp7l\" (UniqueName: \"kubernetes.io/projected/b5c18fbc-be27-4f86-ad76-6a40964d02f9-kube-api-access-4lp7l\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.725205 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.725042 2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b5c18fbc-be27-4f86-ad76-6a40964d02f9-online-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.725205 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.725053 2551 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/574ae380-dbb7-441b-af3a-e6cdd4f434f1-online-tls\") on node \"ip-10-0-140-32.ec2.internal\" DevicePath \"\"" Sep 16 12:16:18.880345 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.880258 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf"] Sep 16 12:16:18.883423 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.883396 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-755cffc58f-m9rbf"] Sep 16 12:16:18.900330 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.900300 2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn"] Sep 16 12:16:18.904903 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:18.904875 2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-74bf7fb59d-d6bfn"] Sep 16 12:16:20.716545 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:20.716491 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" path="/var/lib/kubelet/pods/574ae380-dbb7-441b-af3a-e6cdd4f434f1/volumes" Sep 16 12:16:20.717229 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:16:20.717207 2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" path="/var/lib/kubelet/pods/b5c18fbc-be27-4f86-ad76-6a40964d02f9/volumes" Sep 16 12:22:52.349361 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349321 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjh2m/must-gather-rlvmz"] Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349680 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="feast-apply" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349692 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="feast-apply" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349703 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349709 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349718 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349725 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349733 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349738 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349745 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="feast-init" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349750 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="feast-init" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349762 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e009238-d083-4bda-9e2e-3c75dd76f272" containerName="postgres" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349767 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e009238-d083-4bda-9e2e-3c75dd76f272" containerName="postgres" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349774 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61e8a07-0211-42d9-80fd-fc5bf319bb98" containerName="redis" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349779 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61e8a07-0211-42d9-80fd-fc5bf319bb98" containerName="redis" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349785 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="feast-apply" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349790 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="feast-apply" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349796 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349801 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349808 2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="feast-init" Sep 16 12:22:52.349832 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349813 2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="feast-init" Sep 16 12:22:52.350425 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349865 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61e8a07-0211-42d9-80fd-fc5bf319bb98" containerName="redis" Sep 16 12:22:52.350425 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349871 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="online" Sep 16 12:22:52.350425 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349878 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="registry" Sep 16 12:22:52.350425 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349883 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="574ae380-dbb7-441b-af3a-e6cdd4f434f1" containerName="online" Sep 16 12:22:52.350425 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349890 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e009238-d083-4bda-9e2e-3c75dd76f272" containerName="postgres" Sep 16 12:22:52.350425 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.349895 2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c18fbc-be27-4f86-ad76-6a40964d02f9" containerName="registry" Sep 16 12:22:52.352894 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.352877 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.354823 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.354794 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjh2m\"/\"kube-root-ca.crt\"" Sep 16 12:22:52.354823 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.354793 2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mjh2m\"/\"openshift-service-ca.crt\"" Sep 16 12:22:52.355346 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.355326 2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mjh2m\"/\"default-dockercfg-8n8ld\"" Sep 16 12:22:52.358914 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.358893 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjh2m/must-gather-rlvmz"] Sep 16 12:22:52.378180 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.378141 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq875\" (UniqueName: \"kubernetes.io/projected/8757be8f-3c97-4580-9418-e00896d29266-kube-api-access-xq875\") pod \"must-gather-rlvmz\" (UID: \"8757be8f-3c97-4580-9418-e00896d29266\") " pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.378180 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.378177 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8757be8f-3c97-4580-9418-e00896d29266-must-gather-output\") pod \"must-gather-rlvmz\" (UID: \"8757be8f-3c97-4580-9418-e00896d29266\") " pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.479533 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.479461 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq875\" (UniqueName: \"kubernetes.io/projected/8757be8f-3c97-4580-9418-e00896d29266-kube-api-access-xq875\") pod \"must-gather-rlvmz\" (UID: \"8757be8f-3c97-4580-9418-e00896d29266\") " pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.479533 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.479537 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8757be8f-3c97-4580-9418-e00896d29266-must-gather-output\") pod \"must-gather-rlvmz\" (UID: \"8757be8f-3c97-4580-9418-e00896d29266\") " pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.479894 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.479874 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8757be8f-3c97-4580-9418-e00896d29266-must-gather-output\") pod \"must-gather-rlvmz\" (UID: \"8757be8f-3c97-4580-9418-e00896d29266\") " pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.487259 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.487231 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq875\" (UniqueName: \"kubernetes.io/projected/8757be8f-3c97-4580-9418-e00896d29266-kube-api-access-xq875\") pod \"must-gather-rlvmz\" (UID: \"8757be8f-3c97-4580-9418-e00896d29266\") " pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.675321 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.675277 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjh2m/must-gather-rlvmz" Sep 16 12:22:52.944034 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.944006 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjh2m/must-gather-rlvmz"] Sep 16 12:22:52.946605 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:22:52.946575 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8757be8f_3c97_4580_9418_e00896d29266.slice/crio-e0ad83f8366c38d044e337012f7df230add5969a045ad63982ef0adc1da7b518 WatchSource:0}: Error finding container e0ad83f8366c38d044e337012f7df230add5969a045ad63982ef0adc1da7b518: Status 404 returned error can't find the container with id e0ad83f8366c38d044e337012f7df230add5969a045ad63982ef0adc1da7b518 Sep 16 12:22:52.948576 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:52.948558 2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 12:22:53.165436 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:53.165396 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjh2m/must-gather-rlvmz" event={"ID":"8757be8f-3c97-4580-9418-e00896d29266","Type":"ContainerStarted","Data":"e0ad83f8366c38d044e337012f7df230add5969a045ad63982ef0adc1da7b518"} Sep 16 12:22:54.173901 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:54.173858 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjh2m/must-gather-rlvmz" event={"ID":"8757be8f-3c97-4580-9418-e00896d29266","Type":"ContainerStarted","Data":"9b042cb1687585dd18b8587f4df49435686d3ca8d0d5f41e0c74fffffc8d8509"} Sep 16 12:22:54.174406 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:54.173919 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjh2m/must-gather-rlvmz" event={"ID":"8757be8f-3c97-4580-9418-e00896d29266","Type":"ContainerStarted","Data":"a9c635cb2cc9515c22e543be1061136e2556ac891eda8821af2d9b8159aba265"} Sep 16 12:22:55.027403 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:55.027338 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-62ntj_da7c5f35-c4e1-419d-9c26-5e052ec5130c/global-pull-secret-syncer/0.log" Sep 16 12:22:55.195763 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:55.195731 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xrrn5_deb9a0a2-535a-4e0b-8dab-474b0cd2d70f/konnectivity-agent/0.log" Sep 16 12:22:55.243377 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:55.243344 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-32.ec2.internal_956dcc432e8e3037c236189ed922ba03/haproxy/0.log" Sep 16 12:22:59.146313 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.146286 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/alertmanager/0.log" Sep 16 12:22:59.179146 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.179109 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/config-reloader/0.log" Sep 16 12:22:59.213121 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.213089 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/kube-rbac-proxy-web/0.log" Sep 16 12:22:59.242857 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.242817 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/kube-rbac-proxy/0.log" Sep 16 12:22:59.278957 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.278926 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/kube-rbac-proxy-metric/0.log" Sep 16 12:22:59.303431 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.303396 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/prom-label-proxy/0.log" Sep 16 12:22:59.332592 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.332543 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2310e3d5-4f76-49af-accb-5abd0da241a0/init-config-reloader/0.log" Sep 16 12:22:59.400269 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.400121 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-nvzhm_99f21fd7-3079-4913-827d-8924b3675bee/kube-state-metrics/0.log" Sep 16 12:22:59.426447 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.426419 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-nvzhm_99f21fd7-3079-4913-827d-8924b3675bee/kube-rbac-proxy-main/0.log" Sep 16 12:22:59.450626 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.450596 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-nvzhm_99f21fd7-3079-4913-827d-8924b3675bee/kube-rbac-proxy-self/0.log" Sep 16 12:22:59.492301 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.492274 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5bd97f8f7d-mvmds_7e45b644-e98e-43ca-bf98-a370478cd7fa/metrics-server/0.log" Sep 16 12:22:59.522158 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.522127 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-46jrr_fbd818eb-3f3d-4ae1-999c-69d6817836fc/monitoring-plugin/0.log" Sep 16 12:22:59.728939 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.728908 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xbmg_d2fdbec5-569e-4c65-a0ae-d69c45a23132/node-exporter/0.log" Sep 16 12:22:59.753721 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.753662 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xbmg_d2fdbec5-569e-4c65-a0ae-d69c45a23132/kube-rbac-proxy/0.log" Sep 16 12:22:59.778135 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.778109 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6xbmg_d2fdbec5-569e-4c65-a0ae-d69c45a23132/init-textfile/0.log" Sep 16 12:22:59.810264 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.810232 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-77979_64c36bc7-5406-4ad4-b595-5cb154d913f9/kube-rbac-proxy-main/0.log" Sep 16 12:22:59.835729 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.835703 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-77979_64c36bc7-5406-4ad4-b595-5cb154d913f9/kube-rbac-proxy-self/0.log" Sep 16 12:22:59.867562 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.867529 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-77979_64c36bc7-5406-4ad4-b595-5cb154d913f9/openshift-state-metrics/0.log" Sep 16 12:22:59.909571 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.909539 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/prometheus/0.log" Sep 16 12:22:59.943070 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.942941 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/config-reloader/0.log" Sep 16 12:22:59.972545 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:22:59.972408 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/thanos-sidecar/0.log" Sep 16 12:23:00.000524 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.000447 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/kube-rbac-proxy-web/0.log" Sep 16 12:23:00.028441 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.028398 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/kube-rbac-proxy/0.log" Sep 16 12:23:00.051920 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.051891 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/kube-rbac-proxy-thanos/0.log" Sep 16 12:23:00.077666 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.077590 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95b9ef88-fe9f-4414-aa17-8466e27be6b0/init-config-reloader/0.log" Sep 16 12:23:00.111025 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.110990 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-s6kmp_5be80290-74ba-4c2e-9217-89f746883d65/prometheus-operator/0.log" Sep 16 12:23:00.138439 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.138392 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-s6kmp_5be80290-74ba-4c2e-9217-89f746883d65/kube-rbac-proxy/0.log" Sep 16 12:23:00.165775 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.165744 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-nxwl2_eb2e38f3-42ea-41e9-8a61-791f7e8d014d/prometheus-operator-admission-webhook/0.log" Sep 16 12:23:00.274529 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.274427 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65774677d-bmjp9_cd7dfcfb-7551-483b-a213-c46a3a87aabc/thanos-query/0.log" Sep 16 12:23:00.302655 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.302621 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65774677d-bmjp9_cd7dfcfb-7551-483b-a213-c46a3a87aabc/kube-rbac-proxy-web/0.log" Sep 16 12:23:00.331887 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.331796 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65774677d-bmjp9_cd7dfcfb-7551-483b-a213-c46a3a87aabc/kube-rbac-proxy/0.log" Sep 16 12:23:00.356984 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.356945 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65774677d-bmjp9_cd7dfcfb-7551-483b-a213-c46a3a87aabc/prom-label-proxy/0.log" Sep 16 12:23:00.385207 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.385177 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65774677d-bmjp9_cd7dfcfb-7551-483b-a213-c46a3a87aabc/kube-rbac-proxy-rules/0.log" Sep 16 12:23:00.410284 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:00.410256 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-65774677d-bmjp9_cd7dfcfb-7551-483b-a213-c46a3a87aabc/kube-rbac-proxy-metrics/0.log" Sep 16 12:23:02.588784 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.588757 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-rpbrf_a3b72146-3cfe-499d-ac88-3e6ec0dec451/download-server/0.log" Sep 16 12:23:02.876132 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.876010 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjh2m/must-gather-rlvmz" podStartSLOduration=10.107142363 podStartE2EDuration="10.875987918s" totalImagesPullingTime="768.845555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:54.191606095 +0000 UTC m=+1659.976879665" watchObservedRunningTime="2026-09-16 12:23:02.875987918 +0000 UTC m=+1668.661261488" Sep 16 12:23:02.877843 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.877808 2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl"] Sep 16 12:23:02.883563 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.883534 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:02.891859 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.891827 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl"] Sep 16 12:23:02.982975 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.982935 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-podres\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:02.982975 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.982978 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-lib-modules\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:02.983197 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.983060 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-proc\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:02.983197 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.983121 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54r8n\" (UniqueName: \"kubernetes.io/projected/ab715483-5381-4995-a01d-b314a21e94e8-kube-api-access-54r8n\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:02.983197 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:02.983177 2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-sys\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.083647 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083610 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54r8n\" (UniqueName: \"kubernetes.io/projected/ab715483-5381-4995-a01d-b314a21e94e8-kube-api-access-54r8n\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.083825 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083692 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-sys\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.083825 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083717 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-podres\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.083825 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083737 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-lib-modules\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.083825 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083789 2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-proc\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.084015 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083820 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-sys\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.084015 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083873 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-proc\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.084015 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083899 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-podres\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.084015 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.083932 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab715483-5381-4995-a01d-b314a21e94e8-lib-modules\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.092270 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.092235 2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54r8n\" (UniqueName: \"kubernetes.io/projected/ab715483-5381-4995-a01d-b314a21e94e8-kube-api-access-54r8n\") pod \"perf-node-gather-daemonset-d2xvl\" (UID: \"ab715483-5381-4995-a01d-b314a21e94e8\") " pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.197736 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.197703 2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:03.476285 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.476087 2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl"] Sep 16 12:23:03.479260 ip-10-0-140-32 kubenswrapper[2551]: W0916 12:23:03.479221 2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podab715483_5381_4995_a01d_b314a21e94e8.slice/crio-c9fa8fa792bceaa98dd537fb5791a12f289348ae32b50afc21281afa09a783ea WatchSource:0}: Error finding container c9fa8fa792bceaa98dd537fb5791a12f289348ae32b50afc21281afa09a783ea: Status 404 returned error can't find the container with id c9fa8fa792bceaa98dd537fb5791a12f289348ae32b50afc21281afa09a783ea Sep 16 12:23:03.915067 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.915042 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kzcms_b2833af4-8b90-4c8e-b710-03ca1f39c373/dns/0.log" Sep 16 12:23:03.936138 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:03.936104 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kzcms_b2833af4-8b90-4c8e-b710-03ca1f39c373/kube-rbac-proxy/0.log" Sep 16 12:23:04.081107 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:04.081073 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sdmwq_1843d7ee-d758-4be5-9b19-b1de84c8ff62/dns-node-resolver/0.log" Sep 16 12:23:04.232228 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:04.232131 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" event={"ID":"ab715483-5381-4995-a01d-b314a21e94e8","Type":"ContainerStarted","Data":"9e8db9856d95ef6c616239fed84c6c4911c28701c01704806edeaa98e44aa93e"} Sep 16 12:23:04.232228 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:04.232184 2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" event={"ID":"ab715483-5381-4995-a01d-b314a21e94e8","Type":"ContainerStarted","Data":"c9fa8fa792bceaa98dd537fb5791a12f289348ae32b50afc21281afa09a783ea"} Sep 16 12:23:04.232228 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:04.232210 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:04.247961 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:04.247898 2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" podStartSLOduration=2.247873158 podStartE2EDuration="2.247873158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:04.245832705 +0000 UTC m=+1670.031106273" watchObservedRunningTime="2026-09-16 12:23:04.247873158 +0000 UTC m=+1670.033146727" Sep 16 12:23:04.514869 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:04.514790 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvrwb_d42e00ef-3df2-426e-8ab9-c55b73057716/node-ca/0.log" Sep 16 12:23:05.479588 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:05.479556 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7dz4b_94210cd9-b4b4-42fe-931d-6ddea66155ee/serve-healthcheck-canary/0.log" Sep 16 12:23:05.979286 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:05.979252 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krb84_288a1af8-0e73-4cd8-af08-6533e3ef390b/kube-rbac-proxy/0.log" Sep 16 12:23:05.999872 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:05.999842 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krb84_288a1af8-0e73-4cd8-af08-6533e3ef390b/exporter/0.log" Sep 16 12:23:06.020574 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:06.020551 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-krb84_288a1af8-0e73-4cd8-af08-6533e3ef390b/extractor/0.log" Sep 16 12:23:10.247736 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:10.247702 2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mjh2m/perf-node-gather-daemonset-d2xvl" Sep 16 12:23:11.481036 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.481005 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/kube-multus-additional-cni-plugins/0.log" Sep 16 12:23:11.504543 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.504517 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/egress-router-binary-copy/0.log" Sep 16 12:23:11.532075 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.531994 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/cni-plugins/0.log" Sep 16 12:23:11.557721 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.557694 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/bond-cni-plugin/0.log" Sep 16 12:23:11.582109 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.582085 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/routeoverride-cni/0.log" Sep 16 12:23:11.603709 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.603672 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/whereabouts-cni-bincopy/0.log" Sep 16 12:23:11.626870 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.626833 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d2jbr_1962a963-727e-4a95-95fd-023b72e6eb48/whereabouts-cni/0.log" Sep 16 12:23:11.986610 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:11.986575 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l2zhk_968602c3-6ae7-49f7-9c67-2770f01777d6/kube-multus/0.log" Sep 16 12:23:12.066909 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:12.066863 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-76jr7_272abf77-006f-4a6d-933d-de30d31a1cf5/network-metrics-daemon/0.log" Sep 16 12:23:12.090172 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:12.090143 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-76jr7_272abf77-006f-4a6d-933d-de30d31a1cf5/kube-rbac-proxy/0.log" Sep 16 12:23:13.290751 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.290719 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/ovn-controller/0.log" Sep 16 12:23:13.311296 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.311270 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/ovn-acl-logging/0.log" Sep 16 12:23:13.337658 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.337622 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/kube-rbac-proxy-node/0.log" Sep 16 12:23:13.360707 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.360673 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 12:23:13.382296 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.382265 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/northd/0.log" Sep 16 12:23:13.405661 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.405625 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/nbdb/0.log" Sep 16 12:23:13.429364 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.429339 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/sbdb/0.log" Sep 16 12:23:13.469884 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:13.469835 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjpll_0a8d0d18-4cc5-468d-9aea-a3612523ef6b/ovnkube-controller/0.log" Sep 16 12:23:14.450765 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:14.450733 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mvtdz_8582a7de-79bf-418f-9925-6057c581cf49/network-check-target-container/0.log" Sep 16 12:23:15.320789 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:15.320759 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-gx9kz_23cde59d-a1a1-4063-b447-4e5300a0f09e/iptables-alerter/0.log" Sep 16 12:23:15.920521 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:15.920467 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-kzlbn_5ba2517d-7a25-4d31-b469-d4f9c982d219/tuned/0.log" Sep 16 12:23:19.200155 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:19.200108 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wn2w8_38244eb8-3202-4a13-8e7e-aea8304450c0/csi-driver/0.log" Sep 16 12:23:19.222187 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:19.222156 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wn2w8_38244eb8-3202-4a13-8e7e-aea8304450c0/csi-node-driver-registrar/0.log" Sep 16 12:23:19.243914 ip-10-0-140-32 kubenswrapper[2551]: I0916 12:23:19.243887 2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-wn2w8_38244eb8-3202-4a13-8e7e-aea8304450c0/csi-liveness-probe/0.log"