Sep 15 16:24:46.523933 ip-10-0-142-196 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 15 16:24:46.523949 ip-10-0-142-196 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 15 16:24:46.523959 ip-10-0-142-196 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 15 16:24:46.524171 ip-10-0-142-196 systemd[1]: Failed to start Kubernetes Kubelet. Sep 15 16:24:56.747506 ip-10-0-142-196 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 15 16:24:56.747524 ip-10-0-142-196 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3a852123078f40988344e723a1c79ed5 -- Sep 15 16:26:43.115998 ip-10-0-142-196 systemd[1]: Starting Kubernetes Kubelet... Sep 15 16:26:43.510514 ip-10-0-142-196 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 16:26:43.510514 ip-10-0-142-196 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 16:26:43.510514 ip-10-0-142-196 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 16:26:43.510514 ip-10-0-142-196 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 16:26:43.510514 ip-10-0-142-196 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 16:26:43.511213 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.510572 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 16:26:43.514630 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514613 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 16:26:43.514630 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514629 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514633 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514636 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514642 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514645 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514650 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514653 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514655 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514658 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514661 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514664 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514667 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514669 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514672 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514674 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514677 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514680 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514682 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514685 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 16:26:43.514694 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514689 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514693 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514696 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514699 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514702 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514705 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514708 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514711 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514714 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514717 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514719 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514735 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514738 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514741 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514743 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514746 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514749 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514754 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514757 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 16:26:43.515188 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514759 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514762 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514764 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514767 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514769 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514772 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514774 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514777 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514779 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514783 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514786 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514788 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514790 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514793 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514796 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514800 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514803 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514805 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514808 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514811 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 16:26:43.515643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514813 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514816 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514818 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514821 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514825 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514828 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514831 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514834 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514836 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514839 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514842 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514846 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514849 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514851 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514854 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514857 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514859 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514862 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514865 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 16:26:43.516156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514867 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514870 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514872 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514875 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514878 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514880 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514883 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514886 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514889 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514892 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514894 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514897 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514899 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514902 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514904 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514907 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514909 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514912 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514914 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514917 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.514919 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 16:26:43.516651 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515639 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515647 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515651 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515654 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515658 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515661 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515665 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515668 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515671 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515673 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515676 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515678 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515681 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515684 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515687 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515689 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515692 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515695 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515698 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515701 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 16:26:43.517167 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515704 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515706 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515709 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515712 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515715 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515718 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515732 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515735 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515738 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515741 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515743 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515746 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515749 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515753 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515757 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515760 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515763 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515766 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515769 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 16:26:43.517766 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515772 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515774 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515777 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515781 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515784 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515787 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515791 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515794 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515796 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515798 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515801 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515804 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515807 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515810 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515812 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515815 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515818 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515820 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515824 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 16:26:43.518235 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515828 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515831 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515834 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515836 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515839 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515842 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515845 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515848 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515851 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515853 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515856 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515860 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515863 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515866 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515868 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515871 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515873 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515876 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515878 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515881 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 16:26:43.518700 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515883 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515886 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515888 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515891 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515894 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515897 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515900 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515903 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515905 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515908 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515911 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515913 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515916 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515918 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515921 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515923 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515926 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515928 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515931 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515933 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 16:26:43.519227 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.515936 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516290 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516296 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516299 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516303 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516305 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516308 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516311 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516314 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516317 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516319 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516322 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516325 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516339 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516343 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516347 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516350 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516353 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516356 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516359 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 16:26:43.519715 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516361 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516364 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516367 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516369 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516372 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516374 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516377 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516379 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516382 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516384 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516387 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516389 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516391 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516394 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516397 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516399 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516402 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516405 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516408 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516410 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 16:26:43.520221 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516413 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516415 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516418 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516420 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516423 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516425 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516428 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516431 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516434 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516438 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516441 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516443 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516446 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516448 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516451 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516453 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516456 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516458 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516461 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516463 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 16:26:43.520713 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516465 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516468 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516470 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516473 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516475 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516479 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516483 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516486 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516489 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516491 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516494 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516497 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516500 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516502 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516505 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516507 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516510 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516512 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516516 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 16:26:43.521239 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516520 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516523 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516529 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516532 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516535 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516538 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516540 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516543 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516546 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516548 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516551 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516553 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516556 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516558 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516561 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516564 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516566 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516569 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516571 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 16:26:43.521765 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516574 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.516576 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516645 2558 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516689 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516697 2558 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516702 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516706 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516709 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516718 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516735 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516738 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516741 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516747 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516750 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516755 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516759 2558 flags.go:64] FLAG: --cgroup-root="" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516761 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516764 2558 flags.go:64] FLAG: --client-ca-file="" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516767 2558 flags.go:64] FLAG: --cloud-provider="external" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516770 2558 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516774 2558 flags.go:64] FLAG: --cluster-domain="" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516777 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516780 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516783 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 16:26:43.522219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516786 2558 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516790 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516792 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516795 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516798 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516801 2558 flags.go:64] FLAG: --contention-profiling="false" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516804 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516807 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516810 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516813 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516817 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516820 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516823 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516826 2558 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516833 2558 flags.go:64] FLAG: --enable-server="true" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516836 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516841 2558 flags.go:64] FLAG: --event-burst="100" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516844 2558 flags.go:64] FLAG: --event-qps="50" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516847 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516850 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516856 2558 flags.go:64] FLAG: --eviction-hard="" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516863 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516867 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516870 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516873 2558 flags.go:64] FLAG: --eviction-soft="" Sep 15 16:26:43.522806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516876 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516879 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516882 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516886 2558 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516888 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516891 2558 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516894 2558 flags.go:64] FLAG: --feature-gates="" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516897 2558 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516900 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516903 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516906 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516909 2558 flags.go:64] FLAG: --healthz-port="10248" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516913 2558 flags.go:64] FLAG: --help="false" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516916 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516919 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516921 2558 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516924 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516927 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516930 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516933 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516936 2558 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516939 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516942 2558 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 16:26:43.523416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516945 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516948 2558 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516951 2558 flags.go:64] FLAG: --kube-reserved="" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516953 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516958 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516961 2558 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516965 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516968 2558 flags.go:64] FLAG: --lock-file="" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516971 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516974 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516977 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516982 2558 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516985 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516988 2558 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516991 2558 flags.go:64] FLAG: --logging-format="text" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516993 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.516997 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517000 2558 flags.go:64] FLAG: --manifest-url="" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517002 2558 flags.go:64] FLAG: --manifest-url-header="" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517007 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517009 2558 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517013 2558 flags.go:64] FLAG: --max-pods="110" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517016 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517019 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517022 2558 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 16:26:43.524031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517025 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517028 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517031 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517034 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517041 2558 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517044 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517047 2558 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517053 2558 flags.go:64] FLAG: --pod-cidr="" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517056 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517061 2558 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517064 2558 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517067 2558 flags.go:64] FLAG: --pods-per-core="0" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517071 2558 flags.go:64] FLAG: --port="10250" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517076 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517078 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f511af33fe736c98" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517082 2558 flags.go:64] FLAG: --qos-reserved="" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517085 2558 flags.go:64] FLAG: --read-only-port="10255" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517099 2558 flags.go:64] FLAG: --register-node="true" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517103 2558 flags.go:64] FLAG: --register-with-taints="" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517107 2558 flags.go:64] FLAG: --registry-burst="10" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517110 2558 flags.go:64] FLAG: --registry-qps="5" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517113 2558 flags.go:64] FLAG: --reserved-cpus="" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517116 2558 flags.go:64] FLAG: --reserved-memory="" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517120 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517123 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 16:26:43.524652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517126 2558 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517129 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517132 2558 flags.go:64] FLAG: --runonce="false" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517135 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517138 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517140 2558 flags.go:64] FLAG: --seccomp-default="false" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517143 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517146 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517149 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517152 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517155 2558 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517158 2558 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517161 2558 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517163 2558 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517166 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517170 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517173 2558 flags.go:64] FLAG: --system-cgroups="" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517176 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517181 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517185 2558 flags.go:64] FLAG: --tls-cert-file="" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517189 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517192 2558 flags.go:64] FLAG: --tls-min-version="" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517195 2558 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517198 2558 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517201 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 16:26:43.525428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517204 2558 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517207 2558 flags.go:64] FLAG: --v="2" Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517211 2558 flags.go:64] FLAG: --version="false" Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517214 2558 flags.go:64] FLAG: --vmodule="" Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517218 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.517221 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517328 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517332 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517335 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517338 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517340 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517343 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517345 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517348 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517351 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517353 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517355 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517358 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517360 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517363 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517365 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 16:26:43.526430 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517368 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517371 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517374 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517376 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517379 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517382 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517386 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517389 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517391 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517394 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517396 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517399 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517401 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517404 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517406 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517409 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517411 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517414 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517416 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517419 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 16:26:43.527019 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517422 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517424 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517427 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517429 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517432 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517434 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517437 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517439 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517442 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517445 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517447 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517450 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517452 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517455 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517457 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517461 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517463 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517468 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517473 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517476 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 16:26:43.527519 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517479 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517482 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517485 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517487 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517490 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517492 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517495 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517497 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517500 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517502 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517505 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517508 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517512 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517515 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517518 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517521 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517524 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517526 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517528 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517531 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 16:26:43.528032 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517534 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517536 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517539 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517541 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517544 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517546 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517549 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517551 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517554 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517557 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517560 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517564 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517567 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517570 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517573 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517575 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517578 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517581 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517583 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 16:26:43.528526 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517586 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517589 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517591 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517594 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.517596 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.518168 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.527125 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 16:26:43.529056 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.527140 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 16:26:43.530089 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.530077 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 16:26:43.530123 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.530089 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 16:26:43.530156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530146 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 16:26:43.530156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530152 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 16:26:43.530156 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530155 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530158 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530161 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530164 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530167 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530170 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530173 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530176 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530179 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530182 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530184 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530187 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530189 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530192 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530195 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530198 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530200 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530203 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530206 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530209 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 16:26:43.530234 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530211 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530215 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530217 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530220 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530222 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530225 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530228 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530230 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530233 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530236 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530239 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530242 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530245 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530247 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530250 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530253 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530255 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530258 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530260 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 16:26:43.530747 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530263 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530266 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530269 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530271 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530274 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530285 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530288 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530290 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530293 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530296 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530298 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530301 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530304 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530307 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530311 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530316 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530319 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530322 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530325 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530328 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 16:26:43.531216 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530331 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530333 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530336 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530338 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530341 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530344 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530346 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530349 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530351 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530354 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530357 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530359 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530362 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530364 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530367 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530370 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530372 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530375 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530383 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 16:26:43.531701 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530387 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530392 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530396 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530399 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530402 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530405 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530408 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530411 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530414 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530416 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530419 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530422 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530425 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530427 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530430 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530433 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530435 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530438 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 16:26:43.532174 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530440 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.530445 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530577 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530583 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530586 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530588 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530591 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530594 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530596 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530599 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530601 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530604 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530607 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530610 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530614 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 16:26:43.532608 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530617 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530620 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530622 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530625 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530628 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530630 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530633 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530636 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530639 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530642 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530644 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530647 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530649 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530652 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530655 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530658 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530660 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530663 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530665 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530668 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 16:26:43.533087 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530672 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530675 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530678 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530681 2558 feature_gate.go:328] unrecognized feature gate: Example Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530685 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530689 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530692 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530694 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530697 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530699 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530702 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530705 2558 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530708 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530711 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530715 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530718 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530738 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530742 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530744 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 16:26:43.533583 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530747 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530750 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530753 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530756 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530759 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530761 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530764 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530767 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530769 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530772 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530774 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530777 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530779 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530782 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530784 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530787 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530790 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530792 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530795 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530797 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 16:26:43.534076 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530800 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530802 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530805 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530808 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530811 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530813 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530817 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530820 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530822 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530825 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530827 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530831 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530834 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530836 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530839 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530841 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530844 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530847 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530849 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 16:26:43.534553 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530852 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530855 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530857 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530860 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530862 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530865 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530868 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:43.530870 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.530875 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.530991 2558 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 16:26:43.535089 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.534842 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 16:26:43.535709 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.535697 2558 server.go:1019] "Starting client certificate rotation" Sep 15 16:26:43.535765 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.535751 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 16:26:43.535809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.535784 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 16:26:43.538031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.538018 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 16:26:43.540442 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.540425 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 16:26:43.549834 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.549816 2558 log.go:25] "Validated CRI v1 runtime API" Sep 15 16:26:43.554170 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.554152 2558 log.go:25] "Validated CRI v1 image API" Sep 15 16:26:43.555868 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.555851 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 16:26:43.559617 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.559597 2558 fs.go:135] Filesystem UUIDs: map[53d17579-421c-4a01-b895-59832cfcb6f4:/dev/nvme0n1p4 751c2803-13e3-482c-b71f-924472419da6:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 15 16:26:43.559676 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.559619 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 16:26:43.565434 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.565317 2558 manager.go:216] Machine: {Timestamp:2026-09-15 16:26:43.563829852 +0000 UTC m=+0.333111674 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106212 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f11246f363bc915133b63c258be4 SystemUUID:ec24f112-46f3-63bc-9151-33b63c258be4 BootID:3a852123-078f-4098-8344-e723a1c79ed5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:46:8b:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:46:8b:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:d7:3a:9e:9c:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 16:26:43.565434 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.565425 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 16:26:43.565568 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.565536 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 16:26:43.566637 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.566623 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 16:26:43.568144 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.568123 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 16:26:43.568281 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.568149 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-196.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 16:26:43.569322 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.569312 2558 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 16:26:43.569358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.569326 2558 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 16:26:43.569358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.569334 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 16:26:43.570085 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.570075 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 16:26:43.570813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.570805 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 16:26:43.574341 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574330 2558 state_mem.go:36] "Initialized new in-memory state store" Sep 15 16:26:43.574455 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574445 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 16:26:43.574483 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574476 2558 kubelet.go:486] "Attempting to sync node with API server" Sep 15 16:26:43.574510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574485 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 16:26:43.574510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574495 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 16:26:43.574510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574504 2558 kubelet.go:398] "Adding apiserver pod source" Sep 15 16:26:43.574591 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574516 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 16:26:43.574885 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574872 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 16:26:43.574954 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.574894 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 16:26:43.576818 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.576796 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 16:26:43.580211 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.580196 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 16:26:43.580271 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.580226 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 16:26:43.581888 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581875 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 16:26:43.581945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581898 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 16:26:43.581945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581908 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 16:26:43.581945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581917 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 16:26:43.581945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581931 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 16:26:43.581945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581942 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 16:26:43.582075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581961 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 16:26:43.582075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581970 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 16:26:43.582075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581979 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 16:26:43.582075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.581987 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 16:26:43.582075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.582009 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 16:26:43.582075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.582017 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 16:26:43.583432 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.583415 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 16:26:43.583432 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.583431 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 16:26:43.586061 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.586044 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-196.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 16:26:43.586532 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.586512 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 16:26:43.586578 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.586512 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 16:26:43.587243 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.587231 2558 server.go:1268] "Started kubelet" Sep 15 16:26:43.587325 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.587301 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 16:26:43.587366 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.587311 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 16:26:43.587399 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.587380 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 16:26:43.588107 ip-10-0-142-196 systemd[1]: Started Kubernetes Kubelet. Sep 15 16:26:43.589546 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.589530 2558 server.go:310] "Adding debug handlers to kubelet server" Sep 15 16:26:43.589655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.589551 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 16:26:43.592268 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.592240 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 16:26:43.593415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.593403 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 16:26:43.593415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.593410 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 16:26:43.593544 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.593489 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:43.594065 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594047 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 16:26:43.594139 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594069 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 16:26:43.594139 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594051 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 16:26:43.594216 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594164 2558 reconstruct.go:97] "Volume reconstruction finished" Sep 15 16:26:43.594216 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594171 2558 reconciler.go:29] "Reconciler: start to sync state" Sep 15 16:26:43.594557 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594538 2558 factory.go:55] Registering systemd factory Sep 15 16:26:43.594557 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594557 2558 factory.go:223] Registration of the systemd container factory successfully Sep 15 16:26:43.594919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594818 2558 factory.go:153] Registering CRI-O factory Sep 15 16:26:43.594919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594832 2558 factory.go:223] Registration of the crio container factory successfully Sep 15 16:26:43.594919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594880 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 16:26:43.594919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594897 2558 factory.go:103] Registering Raw factory Sep 15 16:26:43.594919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.594910 2558 manager.go:1196] Started watching for new ooms in manager Sep 15 16:26:43.595243 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.595229 2558 manager.go:349] Starting recovery of all containers Sep 15 16:26:43.603818 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.603793 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 15 16:26:43.604321 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.603799 2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-196.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 15 16:26:43.605984 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.605943 2558 manager.go:354] Recovery completed Sep 15 16:26:43.607229 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.606255 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-196.ec2.internal.18d58b503711ea29 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-196.ec2.internal,UID:ip-10-0-142-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-196.ec2.internal,},FirstTimestamp:2026-09-15 16:26:43.587205673 +0000 UTC m=+0.356487491,LastTimestamp:2026-09-15 16:26:43.587205673 +0000 UTC m=+0.356487491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-196.ec2.internal,}" Sep 15 16:26:43.611612 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.611596 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.614431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.614412 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.614517 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.614448 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.614517 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.614463 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.615044 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615030 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 16:26:43.615044 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615043 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 16:26:43.615161 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615062 2558 state_mem.go:36] "Initialized new in-memory state store" Sep 15 16:26:43.615161 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615071 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.615676 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615663 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.615737 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615689 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.615737 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.615702 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.617018 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.616943 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-196.ec2.internal.18d58b5038b15ab0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-196.ec2.internal,UID:ip-10-0-142-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-196.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-196.ec2.internal,},FirstTimestamp:2026-09-15 16:26:43.61443192 +0000 UTC m=+0.383713738,LastTimestamp:2026-09-15 16:26:43.61443192 +0000 UTC m=+0.383713738,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-196.ec2.internal,}" Sep 15 16:26:43.617166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.617153 2558 policy_none.go:49] "None policy: Start" Sep 15 16:26:43.617677 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.617659 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 16:26:43.617763 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.617684 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 16:26:43.619416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.619397 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85fnx" Sep 15 16:26:43.619708 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.619697 2558 policy_none.go:47] "Start" Sep 15 16:26:43.626909 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.626840 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-196.ec2.internal.18d58b5038b1b32e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-196.ec2.internal,UID:ip-10-0-142-196.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-196.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-196.ec2.internal,},FirstTimestamp:2026-09-15 16:26:43.614454574 +0000 UTC m=+0.383736394,LastTimestamp:2026-09-15 16:26:43.614454574 +0000 UTC m=+0.383736394,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-196.ec2.internal,}" Sep 15 16:26:43.627258 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.627240 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85fnx" Sep 15 16:26:43.646379 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646365 2558 manager.go:337] "Starting Device Plugin manager" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.646408 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646421 2558 server.go:87] "Starting device plugin registration server" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646643 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646653 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646849 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646942 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.646952 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.647396 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 16:26:43.672106 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.647445 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:43.720105 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.720079 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 16:26:43.721192 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.721167 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 16:26:43.721256 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.721229 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 16:26:43.721303 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.721262 2558 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 16:26:43.721343 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.721306 2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 16:26:43.723359 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.723339 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 16:26:43.747130 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.747107 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.747985 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.747968 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.748055 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.748002 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.748055 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.748013 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.748055 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.748042 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.756597 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.756584 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.756663 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.756603 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-196.ec2.internal\": node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:43.784044 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.783974 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:43.822065 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.822040 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal"] Sep 15 16:26:43.822153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.822100 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.822854 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.822836 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.822945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.822865 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.822945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.822878 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.824376 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.824364 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.824494 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.824480 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.824536 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.824508 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.825053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.825036 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.825142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.825063 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.825142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.825069 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.825142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.825086 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.825142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.825095 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.825142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.825072 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.826373 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.826357 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.826457 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.826381 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 16:26:43.826968 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.826954 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientMemory" Sep 15 16:26:43.827039 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.826976 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 16:26:43.827039 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.826987 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeHasSufficientPID" Sep 15 16:26:43.859470 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.859449 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-196.ec2.internal\" not found" node="ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.864043 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.864024 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-196.ec2.internal\" not found" node="ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.884647 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.884627 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:43.895643 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.895624 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe9df7a3782f64b64b388867dcd4e48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal\" (UID: \"ffe9df7a3782f64b64b388867dcd4e48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.895754 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.895655 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c412a2301a21d6df633ce7962cca5842-config\") pod \"kube-apiserver-proxy-ip-10-0-142-196.ec2.internal\" (UID: \"c412a2301a21d6df633ce7962cca5842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.895754 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.895681 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ffe9df7a3782f64b64b388867dcd4e48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal\" (UID: \"ffe9df7a3782f64b64b388867dcd4e48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.985068 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:43.985033 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:43.996409 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.996374 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe9df7a3782f64b64b388867dcd4e48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal\" (UID: \"ffe9df7a3782f64b64b388867dcd4e48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.996472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.996414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c412a2301a21d6df633ce7962cca5842-config\") pod \"kube-apiserver-proxy-ip-10-0-142-196.ec2.internal\" (UID: \"c412a2301a21d6df633ce7962cca5842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.996472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.996430 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ffe9df7a3782f64b64b388867dcd4e48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal\" (UID: \"ffe9df7a3782f64b64b388867dcd4e48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.996472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.996462 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffe9df7a3782f64b64b388867dcd4e48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal\" (UID: \"ffe9df7a3782f64b64b388867dcd4e48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.996578 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.996471 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c412a2301a21d6df633ce7962cca5842-config\") pod \"kube-apiserver-proxy-ip-10-0-142-196.ec2.internal\" (UID: \"c412a2301a21d6df633ce7962cca5842\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" Sep 15 16:26:43.996578 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:43.996515 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ffe9df7a3782f64b64b388867dcd4e48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal\" (UID: \"ffe9df7a3782f64b64b388867dcd4e48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:44.085858 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.085782 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:44.162358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.162329 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:44.166707 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.166444 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" Sep 15 16:26:44.186422 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.186298 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:44.292371 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.292333 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:44.400174 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.400088 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:44.500226 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.500199 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-196.ec2.internal\" not found" Sep 15 16:26:44.501060 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.501044 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 16:26:44.537177 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.537143 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 16:26:44.537616 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.537250 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 16:26:44.537616 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.537297 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 16:26:44.575156 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.575136 2558 apiserver.go:52] "Watching apiserver" Sep 15 16:26:44.594616 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.594400 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 16:26:44.594616 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.594595 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" Sep 15 16:26:44.602115 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.602095 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 16:26:44.602499 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.602475 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-gdxnf","openshift-image-registry/node-ca-ct27n","openshift-multus/network-metrics-daemon-wlfzm","openshift-network-diagnostics/network-check-target-vgmpk","openshift-network-operator/iptables-alerter-9ctfv","openshift-ovn-kubernetes/ovnkube-node-rwngp","kube-system/global-pull-secret-syncer-82z9t","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg","openshift-cluster-node-tuning-operator/tuned-crws5","openshift-dns/node-resolver-jsdtg","openshift-multus/multus-additional-cni-plugins-cj4xv","openshift-multus/multus-vj8vc"] Sep 15 16:26:44.605038 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.605018 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.607537 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.607518 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.607806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.607781 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-trxkn\"" Sep 15 16:26:44.607984 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.607966 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 16:26:44.608040 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.608002 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 16:26:44.609922 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.609902 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.610161 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.610137 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.610261 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.610159 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 16:26:44.610261 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.610229 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xpm5p\"" Sep 15 16:26:44.610375 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.610330 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.610656 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.610638 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 16:26:44.612425 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.612408 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.613155 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613136 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 16:26:44.613242 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613171 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.613242 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613143 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 16:26:44.613521 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613502 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-shsr5\"" Sep 15 16:26:44.613620 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613523 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 16:26:44.613620 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613535 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 16:26:44.613620 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.613502 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.615198 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.614660 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.615198 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.614672 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.615198 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.614846 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.615198 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.614971 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4jxq9\"" Sep 15 16:26:44.615936 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.615887 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 16:26:44.616051 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.615968 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" Sep 15 16:26:44.617193 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.617162 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.617618 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.617595 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79pft\"" Sep 15 16:26:44.617712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.617628 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.617712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.617665 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.620197 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.620180 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.620910 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.620891 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 16:26:44.621012 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.620932 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 16:26:44.621077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.621024 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-trdvs\"" Sep 15 16:26:44.621202 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.621182 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.621346 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.621329 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 16:26:44.622592 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.622568 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.623123 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.623107 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.623210 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.623201 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d9cqm\"" Sep 15 16:26:44.623490 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.623472 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.624954 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hdq26\"" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.624977 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.625029 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.625041 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.626022 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.626214 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.626978 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.627523 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:44.628031 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.627591 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:44.631089 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.631064 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.631489 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.631441 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 16:21:44 +0000 UTC" deadline="2028-06-22 21:58:45.869099022 +0000 UTC" Sep 15 16:26:44.631580 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.631482 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:44.631580 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.631490 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15509h32m1.237613339s" Sep 15 16:26:44.637222 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.637203 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal"] Sep 15 16:26:44.637222 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.637231 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal"] Sep 15 16:26:44.637341 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.637313 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.640032 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.640012 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mgtgz\"" Sep 15 16:26:44.640133 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.640036 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 16:26:44.640133 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.640048 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 16:26:44.640133 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.640069 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 16:26:44.644362 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.644344 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8lq2w" Sep 15 16:26:44.655160 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.655110 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8lq2w" Sep 15 16:26:44.695245 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.695228 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 16:26:44.700563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700411 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31f363f1-7083-44b7-b0c9-8ee80b2ded87-konnectivity-ca\") pod \"konnectivity-agent-gdxnf\" (UID: \"31f363f1-7083-44b7-b0c9-8ee80b2ded87\") " pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.700563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700486 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.700563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700512 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cd19c25-3c74-418d-bab8-dc432ccc1e88-host\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.700563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700529 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f22ba54-c674-4533-a54a-7e7315f5562b-cni-binary-copy\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.700563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700544 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-ovnkube-config\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700594 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-systemd\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700633 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cnibin\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700668 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700701 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-etc-selinux\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700727 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-system-cni-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700749 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-cnibin\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700773 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-socket-dir-parent\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700813 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-socket-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700832 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-cni-multus\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.700861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700851 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-kubelet\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-hostroot\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700900 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcrj\" (UniqueName: \"kubernetes.io/projected/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-kube-api-access-kbcrj\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700917 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700931 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-var-lib-kubelet\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.700984 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2mk9\" (UniqueName: \"kubernetes.io/projected/c86ff085-8efe-41aa-8638-f4b36bf5d910-kube-api-access-w2mk9\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701022 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-registration-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701040 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/480f2cdd-804a-4d7f-b865-3ddd4da269c7-host-slash\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701054 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-k8s-cni-cncf-io\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-cni-bin\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701083 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701100 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701127 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-tuned\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701164 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysctl-d\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/0cd19c25-3c74-418d-bab8-dc432ccc1e88-kube-api-access-t9tqj\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701213 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdz8\" (UniqueName: \"kubernetes.io/projected/da83f919-25b3-4d35-872b-d9634c7a1cc3-kube-api-access-2fdz8\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.701338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701230 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/480f2cdd-804a-4d7f-b865-3ddd4da269c7-iptables-alerter-script\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701249 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdfz\" (UniqueName: \"kubernetes.io/projected/480f2cdd-804a-4d7f-b865-3ddd4da269c7-kube-api-access-4vdfz\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701271 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysconfig\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701286 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysctl-conf\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701302 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701340 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0cd19c25-3c74-418d-bab8-dc432ccc1e88-serviceca\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701372 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6720930f-c883-40dc-bea0-11fc91a597c9-kubelet-config\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701400 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-run-netns\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701427 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-env-overrides\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701450 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-run\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701475 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-tmp-dir\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701500 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-device-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701518 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-node-log\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701536 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-cni-bin\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701563 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701580 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-kubernetes\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.701867 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701599 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-sys\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701619 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-lib-modules\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701638 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-system-cni-dir\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701653 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701669 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-systemd-units\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701710 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-slash\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701747 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-ovn\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701777 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-daemon-config\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701798 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2m8\" (UniqueName: \"kubernetes.io/projected/1f22ba54-c674-4533-a54a-7e7315f5562b-kube-api-access-bf2m8\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701837 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-kubelet\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701884 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-etc-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701916 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkfl\" (UniqueName: \"kubernetes.io/projected/04db1029-90cb-4a75-a9c6-c0d1567cb33d-kube-api-access-9nkfl\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701957 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-log-socket\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701975 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-cni-netd\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.701991 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c86ff085-8efe-41aa-8638-f4b36bf5d910-tmp\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.702556 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702009 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-sys-fs\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702042 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-conf-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702099 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6720930f-c883-40dc-bea0-11fc91a597c9-dbus\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702126 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31f363f1-7083-44b7-b0c9-8ee80b2ded87-agent-certs\") pod \"konnectivity-agent-gdxnf\" (UID: \"31f363f1-7083-44b7-b0c9-8ee80b2ded87\") " pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702154 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdxj\" (UniqueName: \"kubernetes.io/projected/f36bf4e7-e893-4d81-8285-66f26193a29a-kube-api-access-ppdxj\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702185 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-modprobe-d\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702229 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-hosts-file\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702248 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-cni-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702331 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-var-lib-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702363 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f36bf4e7-e893-4d81-8285-66f26193a29a-ovn-node-metrics-cert\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702388 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgwvw\" (UniqueName: \"kubernetes.io/projected/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-kube-api-access-qgwvw\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702410 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-netns\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702433 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-systemd\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702457 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-host\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702479 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.703166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702502 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-ovnkube-script-lib\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.703706 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702537 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-multus-certs\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.703706 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.702574 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-etc-kubernetes\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.761376 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.761339 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe9df7a3782f64b64b388867dcd4e48.slice/crio-b95e00aed04fbd95c7987fbd7084752bab62d3aa29b901d797746482b6539973 WatchSource:0}: Error finding container b95e00aed04fbd95c7987fbd7084752bab62d3aa29b901d797746482b6539973: Status 404 returned error can't find the container with id b95e00aed04fbd95c7987fbd7084752bab62d3aa29b901d797746482b6539973 Sep 15 16:26:44.761643 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.761609 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc412a2301a21d6df633ce7962cca5842.slice/crio-f3ab232fbbac8f072d351c706d8c07bb800a5a07ddf511fd0ed543fa83919837 WatchSource:0}: Error finding container f3ab232fbbac8f072d351c706d8c07bb800a5a07ddf511fd0ed543fa83919837: Status 404 returned error can't find the container with id f3ab232fbbac8f072d351c706d8c07bb800a5a07ddf511fd0ed543fa83919837 Sep 15 16:26:44.768328 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.768311 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 16:26:44.802966 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.802941 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysconfig\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.802988 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysctl-conf\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803018 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.803067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803028 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysconfig\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803043 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0cd19c25-3c74-418d-bab8-dc432ccc1e88-serviceca\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.803067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803066 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6720930f-c883-40dc-bea0-11fc91a597c9-kubelet-config\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803090 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-run-netns\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803112 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-env-overrides\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803123 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysctl-conf\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803160 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-run-netns\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803179 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6720930f-c883-40dc-bea0-11fc91a597c9-kubelet-config\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803228 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-run\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803298 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-tmp-dir\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803327 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-device-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803353 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-node-log\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803377 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-cni-bin\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803403 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803430 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-kubernetes\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803452 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-sys\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803473 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-lib-modules\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803496 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-system-cni-dir\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803519 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803544 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-systemd-units\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803562 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0cd19c25-3c74-418d-bab8-dc432ccc1e88-serviceca\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-slash\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803602 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803611 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-slash\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803424 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-run\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.803639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803633 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-ovn\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803661 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-daemon-config\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-device-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803672 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-sys\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803671 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-tmp-dir\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803688 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2m8\" (UniqueName: \"kubernetes.io/projected/1f22ba54-c674-4533-a54a-7e7315f5562b-kube-api-access-bf2m8\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803707 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-node-log\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803708 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-env-overrides\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803717 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803714 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803748 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-system-cni-dir\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803800 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-kubernetes\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803829 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-cni-bin\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803849 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.803890 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-lib-modules\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.804031 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804087 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-ovn\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.804466 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.804100 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:26:45.304080454 +0000 UTC m=+2.073362285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804122 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-kubelet\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804138 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-systemd-units\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804169 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-etc-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804178 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-kubelet\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804207 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkfl\" (UniqueName: \"kubernetes.io/projected/04db1029-90cb-4a75-a9c6-c0d1567cb33d-kube-api-access-9nkfl\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804215 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-etc-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804234 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-log-socket\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804251 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-cni-netd\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804266 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c86ff085-8efe-41aa-8638-f4b36bf5d910-tmp\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804282 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-sys-fs\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804301 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-conf-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804320 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6720930f-c883-40dc-bea0-11fc91a597c9-dbus\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-log-socket\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804357 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-sys-fs\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804345 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31f363f1-7083-44b7-b0c9-8ee80b2ded87-agent-certs\") pod \"konnectivity-agent-gdxnf\" (UID: \"31f363f1-7083-44b7-b0c9-8ee80b2ded87\") " pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804377 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-conf-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804393 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-cni-netd\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.805318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804400 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdxj\" (UniqueName: \"kubernetes.io/projected/f36bf4e7-e893-4d81-8285-66f26193a29a-kube-api-access-ppdxj\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804359 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-daemon-config\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804427 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-modprobe-d\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804454 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-hosts-file\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804465 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6720930f-c883-40dc-bea0-11fc91a597c9-dbus\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804477 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804505 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-cni-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804515 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-hosts-file\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804532 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-var-lib-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804544 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-modprobe-d\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804558 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f36bf4e7-e893-4d81-8285-66f26193a29a-ovn-node-metrics-cert\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804571 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-cni-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804584 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgwvw\" (UniqueName: \"kubernetes.io/projected/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-kube-api-access-qgwvw\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804586 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-var-lib-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-netns\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804634 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-systemd\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804650 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804658 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-host\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.806153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804684 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804691 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-netns\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804711 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-ovnkube-script-lib\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804736 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-multus-certs\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804738 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-systemd\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804780 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-etc-kubernetes\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804791 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-host\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804807 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31f363f1-7083-44b7-b0c9-8ee80b2ded87-konnectivity-ca\") pod \"konnectivity-agent-gdxnf\" (UID: \"31f363f1-7083-44b7-b0c9-8ee80b2ded87\") " pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804836 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804868 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-multus-certs\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804876 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-etc-kubernetes\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804888 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-host-run-ovn-kubernetes\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cd19c25-3c74-418d-bab8-dc432ccc1e88-host\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804937 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f22ba54-c674-4533-a54a-7e7315f5562b-cni-binary-copy\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804952 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.804970 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-ovnkube-config\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805000 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805016 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-systemd\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.807053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805053 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cnibin\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805072 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cd19c25-3c74-418d-bab8-dc432ccc1e88-host\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805078 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805105 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-etc-selinux\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805125 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-system-cni-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-cnibin\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805167 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-socket-dir-parent\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805188 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-socket-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805210 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-cni-multus\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805229 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-kubelet\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805250 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-hostroot\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805271 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcrj\" (UniqueName: \"kubernetes.io/projected/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-kube-api-access-kbcrj\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805292 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805314 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-var-lib-kubelet\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805335 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2mk9\" (UniqueName: \"kubernetes.io/projected/c86ff085-8efe-41aa-8638-f4b36bf5d910-kube-api-access-w2mk9\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805365 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-registration-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805389 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/480f2cdd-804a-4d7f-b865-3ddd4da269c7-host-slash\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.808026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805413 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-k8s-cni-cncf-io\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805436 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-cni-bin\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805463 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805485 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-tuned\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysctl-d\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805531 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31f363f1-7083-44b7-b0c9-8ee80b2ded87-konnectivity-ca\") pod \"konnectivity-agent-gdxnf\" (UID: \"31f363f1-7083-44b7-b0c9-8ee80b2ded87\") " pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805536 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f22ba54-c674-4533-a54a-7e7315f5562b-cni-binary-copy\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805555 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-ovnkube-script-lib\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805580 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/0cd19c25-3c74-418d-bab8-dc432ccc1e88-kube-api-access-t9tqj\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdz8\" (UniqueName: \"kubernetes.io/projected/da83f919-25b3-4d35-872b-d9634c7a1cc3-kube-api-access-2fdz8\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805633 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-sysctl-d\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805655 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/480f2cdd-804a-4d7f-b865-3ddd4da269c7-iptables-alerter-script\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805672 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-systemd\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805686 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdfz\" (UniqueName: \"kubernetes.io/projected/480f2cdd-804a-4d7f-b865-3ddd4da269c7-kube-api-access-4vdfz\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805710 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cnibin\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.805964 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f36bf4e7-e893-4d81-8285-66f26193a29a-ovnkube-config\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.808839 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.806080 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.806149 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:45.306130806 +0000 UTC m=+2.075412631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806173 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04db1029-90cb-4a75-a9c6-c0d1567cb33d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806205 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-system-cni-dir\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-kubelet\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806329 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-cni-multus\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806348 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-socket-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806377 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-multus-socket-dir-parent\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806400 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-run-k8s-cni-cncf-io\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806575 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-registration-dir\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806246 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86ff085-8efe-41aa-8638-f4b36bf5d910-var-lib-kubelet\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806631 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/480f2cdd-804a-4d7f-b865-3ddd4da269c7-host-slash\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806636 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f36bf4e7-e893-4d81-8285-66f26193a29a-run-openvswitch\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806647 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-host-var-lib-cni-bin\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806661 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/480f2cdd-804a-4d7f-b865-3ddd4da269c7-iptables-alerter-script\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806697 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-hostroot\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806715 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f22ba54-c674-4533-a54a-7e7315f5562b-cnibin\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.809438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.806732 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da83f919-25b3-4d35-872b-d9634c7a1cc3-etc-selinux\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.809969 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.808646 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f36bf4e7-e893-4d81-8285-66f26193a29a-ovn-node-metrics-cert\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.809969 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.808757 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c86ff085-8efe-41aa-8638-f4b36bf5d910-tmp\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.809969 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.808975 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c86ff085-8efe-41aa-8638-f4b36bf5d910-etc-tuned\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.809969 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.809005 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31f363f1-7083-44b7-b0c9-8ee80b2ded87-agent-certs\") pod \"konnectivity-agent-gdxnf\" (UID: \"31f363f1-7083-44b7-b0c9-8ee80b2ded87\") " pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.816451 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.815968 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:26:44.816451 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.815991 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:26:44.816451 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.816004 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:44.816451 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:44.816064 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:45.316047841 +0000 UTC m=+2.085329669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:44.816451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.816335 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgwvw\" (UniqueName: \"kubernetes.io/projected/0ed7aa9b-df82-4e1a-b35c-89087c2ced64-kube-api-access-qgwvw\") pod \"node-resolver-jsdtg\" (UID: \"0ed7aa9b-df82-4e1a-b35c-89087c2ced64\") " pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.817273 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.817251 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdxj\" (UniqueName: \"kubernetes.io/projected/f36bf4e7-e893-4d81-8285-66f26193a29a-kube-api-access-ppdxj\") pod \"ovnkube-node-rwngp\" (UID: \"f36bf4e7-e893-4d81-8285-66f26193a29a\") " pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.817437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.817412 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkfl\" (UniqueName: \"kubernetes.io/projected/04db1029-90cb-4a75-a9c6-c0d1567cb33d-kube-api-access-9nkfl\") pod \"multus-additional-cni-plugins-cj4xv\" (UID: \"04db1029-90cb-4a75-a9c6-c0d1567cb33d\") " pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:44.818009 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.817986 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdfz\" (UniqueName: \"kubernetes.io/projected/480f2cdd-804a-4d7f-b865-3ddd4da269c7-kube-api-access-4vdfz\") pod \"iptables-alerter-9ctfv\" (UID: \"480f2cdd-804a-4d7f-b865-3ddd4da269c7\") " pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.818133 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.818111 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdz8\" (UniqueName: \"kubernetes.io/projected/da83f919-25b3-4d35-872b-d9634c7a1cc3-kube-api-access-2fdz8\") pod \"aws-ebs-csi-driver-node-rxmtg\" (UID: \"da83f919-25b3-4d35-872b-d9634c7a1cc3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:44.818489 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.818468 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcrj\" (UniqueName: \"kubernetes.io/projected/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-kube-api-access-kbcrj\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:44.818548 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.818517 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tqj\" (UniqueName: \"kubernetes.io/projected/0cd19c25-3c74-418d-bab8-dc432ccc1e88-kube-api-access-t9tqj\") pod \"node-ca-ct27n\" (UID: \"0cd19c25-3c74-418d-bab8-dc432ccc1e88\") " pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:44.819120 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.819105 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2mk9\" (UniqueName: \"kubernetes.io/projected/c86ff085-8efe-41aa-8638-f4b36bf5d910-kube-api-access-w2mk9\") pod \"tuned-crws5\" (UID: \"c86ff085-8efe-41aa-8638-f4b36bf5d910\") " pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.819408 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.819394 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2m8\" (UniqueName: \"kubernetes.io/projected/1f22ba54-c674-4533-a54a-7e7315f5562b-kube-api-access-bf2m8\") pod \"multus-vj8vc\" (UID: \"1f22ba54-c674-4533-a54a-7e7315f5562b\") " pod="openshift-multus/multus-vj8vc" Sep 15 16:26:44.842351 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.842324 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 16:26:44.922775 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.922695 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:26:44.933460 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.933437 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f363f1_7083_44b7_b0c9_8ee80b2ded87.slice/crio-9b6ccbea10df1ba5c7a73ba16088467966909ce6b638ebb50f101b40a0749e94 WatchSource:0}: Error finding container 9b6ccbea10df1ba5c7a73ba16088467966909ce6b638ebb50f101b40a0749e94: Status 404 returned error can't find the container with id 9b6ccbea10df1ba5c7a73ba16088467966909ce6b638ebb50f101b40a0749e94 Sep 15 16:26:44.941658 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.941641 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jsdtg" Sep 15 16:26:44.949870 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.949849 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed7aa9b_df82_4e1a_b35c_89087c2ced64.slice/crio-107d18eda5ebfeca10579375b056558937e6d19c6441e4c21590f3f0a4df63c4 WatchSource:0}: Error finding container 107d18eda5ebfeca10579375b056558937e6d19c6441e4c21590f3f0a4df63c4: Status 404 returned error can't find the container with id 107d18eda5ebfeca10579375b056558937e6d19c6441e4c21590f3f0a4df63c4 Sep 15 16:26:44.957747 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.957727 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9ctfv" Sep 15 16:26:44.961254 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.961237 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 16:26:44.965727 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.965699 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480f2cdd_804a_4d7f_b865_3ddd4da269c7.slice/crio-77c6de16cc00e6c42b9f7eacbd93d27eaa3eaf7708422601c489cbd43cbec516 WatchSource:0}: Error finding container 77c6de16cc00e6c42b9f7eacbd93d27eaa3eaf7708422601c489cbd43cbec516: Status 404 returned error can't find the container with id 77c6de16cc00e6c42b9f7eacbd93d27eaa3eaf7708422601c489cbd43cbec516 Sep 15 16:26:44.968896 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.968877 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:26:44.978370 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.978340 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36bf4e7_e893_4d81_8285_66f26193a29a.slice/crio-962efa79f1df09cf8747591553e20186bc947290dd810efef99cb789ea716a2e WatchSource:0}: Error finding container 962efa79f1df09cf8747591553e20186bc947290dd810efef99cb789ea716a2e: Status 404 returned error can't find the container with id 962efa79f1df09cf8747591553e20186bc947290dd810efef99cb789ea716a2e Sep 15 16:26:44.984579 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.984556 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-crws5" Sep 15 16:26:44.992182 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:44.992161 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86ff085_8efe_41aa_8638_f4b36bf5d910.slice/crio-b44dbe116e457b9ffdfc98f129dea5cf48a0fd16d23c0ff4793658912ce7d48d WatchSource:0}: Error finding container b44dbe116e457b9ffdfc98f129dea5cf48a0fd16d23c0ff4793658912ce7d48d: Status 404 returned error can't find the container with id b44dbe116e457b9ffdfc98f129dea5cf48a0fd16d23c0ff4793658912ce7d48d Sep 15 16:26:44.998555 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:44.998533 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" Sep 15 16:26:45.003589 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.003570 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vj8vc" Sep 15 16:26:45.006632 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:45.006611 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04db1029_90cb_4a75_a9c6_c0d1567cb33d.slice/crio-69415cbe0619141a0ffeeff755f0969a193a4b2bf02d6d2e9c42210bcf6143f4 WatchSource:0}: Error finding container 69415cbe0619141a0ffeeff755f0969a193a4b2bf02d6d2e9c42210bcf6143f4: Status 404 returned error can't find the container with id 69415cbe0619141a0ffeeff755f0969a193a4b2bf02d6d2e9c42210bcf6143f4 Sep 15 16:26:45.011512 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:45.011495 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f22ba54_c674_4533_a54a_7e7315f5562b.slice/crio-59b1f6427934162e96e291325e6701e6c07491720083d4d33a3dbd3f5d5d8099 WatchSource:0}: Error finding container 59b1f6427934162e96e291325e6701e6c07491720083d4d33a3dbd3f5d5d8099: Status 404 returned error can't find the container with id 59b1f6427934162e96e291325e6701e6c07491720083d4d33a3dbd3f5d5d8099 Sep 15 16:26:45.018725 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.018708 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ct27n" Sep 15 16:26:45.022481 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.022457 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" Sep 15 16:26:45.028829 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:45.028808 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd19c25_3c74_418d_bab8_dc432ccc1e88.slice/crio-6025e2019438266c946d59d8ca72c844ec009744c0f0ee664d89a759831b4b50 WatchSource:0}: Error finding container 6025e2019438266c946d59d8ca72c844ec009744c0f0ee664d89a759831b4b50: Status 404 returned error can't find the container with id 6025e2019438266c946d59d8ca72c844ec009744c0f0ee664d89a759831b4b50 Sep 15 16:26:45.034145 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:26:45.034125 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda83f919_25b3_4d35_872b_d9634c7a1cc3.slice/crio-931b5cb1e176d401c790dce3231be59c34938e1ff506134a57a07c03e5b5b5ad WatchSource:0}: Error finding container 931b5cb1e176d401c790dce3231be59c34938e1ff506134a57a07c03e5b5b5ad: Status 404 returned error can't find the container with id 931b5cb1e176d401c790dce3231be59c34938e1ff506134a57a07c03e5b5b5ad Sep 15 16:26:45.308695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.308609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:45.308870 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.308712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:45.308870 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.308790 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:45.308870 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.308861 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:45.309048 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.308870 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:46.308848916 +0000 UTC m=+3.078130723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:45.309048 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.308941 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:26:46.308922341 +0000 UTC m=+3.078204154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:45.409921 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.409872 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:45.410097 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.410062 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:26:45.410097 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.410083 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:26:45.410097 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.410096 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:45.410263 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.410157 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:46.410137998 +0000 UTC m=+3.179419809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:45.658647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.658365 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:45 +0000 UTC" deadline="2028-05-22 11:40:40.584578038 +0000 UTC" Sep 15 16:26:45.658647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.658427 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14755h13m54.926155456s" Sep 15 16:26:45.724910 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.724851 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:45.725094 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:45.725010 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:45.757897 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.757749 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" event={"ID":"c412a2301a21d6df633ce7962cca5842","Type":"ContainerStarted","Data":"f3ab232fbbac8f072d351c706d8c07bb800a5a07ddf511fd0ed543fa83919837"} Sep 15 16:26:45.765071 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.765037 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" event={"ID":"ffe9df7a3782f64b64b388867dcd4e48","Type":"ContainerStarted","Data":"b95e00aed04fbd95c7987fbd7084752bab62d3aa29b901d797746482b6539973"} Sep 15 16:26:45.769654 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.769621 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" event={"ID":"da83f919-25b3-4d35-872b-d9634c7a1cc3","Type":"ContainerStarted","Data":"931b5cb1e176d401c790dce3231be59c34938e1ff506134a57a07c03e5b5b5ad"} Sep 15 16:26:45.791275 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.791234 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ct27n" event={"ID":"0cd19c25-3c74-418d-bab8-dc432ccc1e88","Type":"ContainerStarted","Data":"6025e2019438266c946d59d8ca72c844ec009744c0f0ee664d89a759831b4b50"} Sep 15 16:26:45.802474 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.802396 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerStarted","Data":"69415cbe0619141a0ffeeff755f0969a193a4b2bf02d6d2e9c42210bcf6143f4"} Sep 15 16:26:45.816751 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.816650 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-crws5" event={"ID":"c86ff085-8efe-41aa-8638-f4b36bf5d910","Type":"ContainerStarted","Data":"b44dbe116e457b9ffdfc98f129dea5cf48a0fd16d23c0ff4793658912ce7d48d"} Sep 15 16:26:45.830728 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.830686 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ctfv" event={"ID":"480f2cdd-804a-4d7f-b865-3ddd4da269c7","Type":"ContainerStarted","Data":"77c6de16cc00e6c42b9f7eacbd93d27eaa3eaf7708422601c489cbd43cbec516"} Sep 15 16:26:45.833256 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.833225 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vj8vc" event={"ID":"1f22ba54-c674-4533-a54a-7e7315f5562b","Type":"ContainerStarted","Data":"59b1f6427934162e96e291325e6701e6c07491720083d4d33a3dbd3f5d5d8099"} Sep 15 16:26:45.837536 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.837506 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"962efa79f1df09cf8747591553e20186bc947290dd810efef99cb789ea716a2e"} Sep 15 16:26:45.843223 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.843192 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jsdtg" event={"ID":"0ed7aa9b-df82-4e1a-b35c-89087c2ced64","Type":"ContainerStarted","Data":"107d18eda5ebfeca10579375b056558937e6d19c6441e4c21590f3f0a4df63c4"} Sep 15 16:26:45.848980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:45.848947 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdxnf" event={"ID":"31f363f1-7083-44b7-b0c9-8ee80b2ded87","Type":"ContainerStarted","Data":"9b6ccbea10df1ba5c7a73ba16088467966909ce6b638ebb50f101b40a0749e94"} Sep 15 16:26:46.011753 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.011661 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 16:26:46.315947 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.315858 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:46.315947 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.315941 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:46.316188 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.316076 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:46.316188 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.316143 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:26:48.316122779 +0000 UTC m=+5.085404591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:46.316740 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.316722 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:46.316841 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.316779 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:48.316763278 +0000 UTC m=+5.086045088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:46.417137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.417095 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:46.417340 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.417296 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:26:46.417340 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.417322 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:26:46.417340 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.417335 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:46.417505 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.417399 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:48.417379616 +0000 UTC m=+5.186661440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:46.659352 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.659241 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:45 +0000 UTC" deadline="2028-05-31 15:28:21.675291562 +0000 UTC" Sep 15 16:26:46.659352 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.659287 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14975h1m35.016010244s" Sep 15 16:26:46.721864 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.721834 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:46.722031 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.721960 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:46.722400 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:46.722353 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:46.722521 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:46.722459 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:47.444591 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:47.444353 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 16:26:47.722010 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:47.721924 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:47.722507 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:47.722087 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:48.332808 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:48.332770 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:48.333021 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:48.332857 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:48.333021 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.332978 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:48.333137 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.333041 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.3330232 +0000 UTC m=+9.102305007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:48.333332 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.333306 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:48.333443 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.333373 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.333353662 +0000 UTC m=+9.102635479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:48.433407 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:48.433368 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:48.433598 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.433571 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:26:48.433598 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.433597 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:26:48.433759 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.433611 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:48.433759 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.433682 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.43366205 +0000 UTC m=+9.202943869 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:48.723138 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:48.722428 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:48.723138 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.722583 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:48.723138 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:48.722991 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:48.723138 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:48.723085 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:49.721529 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:49.721490 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:49.721735 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:49.721638 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:50.721876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:50.721835 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:50.722288 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:50.721960 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:50.722288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:50.722026 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:50.722288 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:50.722145 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:51.722543 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:51.722184 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:51.722543 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:51.722351 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:52.370919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:52.370870 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:52.371124 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:52.370948 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:52.371124 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.371007 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:52.371124 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.371060 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:52.371124 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.371068 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:27:00.371049348 +0000 UTC m=+17.140331174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:26:52.371124 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.371097 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:00.371085206 +0000 UTC m=+17.140367014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:26:52.471918 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:52.471296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:52.471918 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.471458 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:26:52.471918 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.471483 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:26:52.471918 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.471496 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:52.471918 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.471555 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:00.471536386 +0000 UTC m=+17.240818195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:26:52.722342 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:52.722260 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:52.722507 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.722413 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:52.722702 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:52.722677 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:52.723154 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:52.722796 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:53.723295 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:53.723260 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:53.723750 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:53.723387 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:54.721911 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:54.721822 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:54.722077 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:54.721961 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:54.722077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:54.721822 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:54.722191 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:54.722105 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:55.722478 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:55.722448 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:55.723004 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:55.722588 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:56.722407 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:56.722374 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:56.722546 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:56.722491 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:56.722546 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:56.722528 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:56.722952 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:56.722600 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:57.722281 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:57.722245 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:57.722466 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:57.722402 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:26:58.721805 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:58.721771 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:26:58.722253 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:58.721818 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:26:58.722253 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:58.721923 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:26:58.722253 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:58.722021 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:26:59.721757 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:26:59.721711 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:26:59.721923 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:26:59.721845 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:00.439587 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:00.439554 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:00.439789 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:00.439600 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:00.439789 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.439695 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:27:00.439789 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.439696 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:27:00.439789 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.439755 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:16.439742978 +0000 UTC m=+33.209024782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:27:00.439789 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.439770 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:27:16.439763008 +0000 UTC m=+33.209044813 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:27:00.539951 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:00.539920 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:00.540085 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.540048 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:27:00.540085 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.540064 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:27:00.540085 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.540072 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:27:00.540218 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.540122 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:16.540107039 +0000 UTC m=+33.309388857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:27:00.722432 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:00.722405 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:00.722826 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:00.722440 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:00.722826 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.722528 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:00.722826 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:00.722617 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:01.722544 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:01.722461 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:01.723016 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:01.722614 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:02.722129 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:02.722088 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:02.722317 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:02.722100 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:02.722317 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:02.722210 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:02.722317 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:02.722262 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:03.724922 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:03.723193 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:03.724922 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:03.723380 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:04.721616 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.721591 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:04.721745 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.721626 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:04.721745 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:04.721685 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:04.721832 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:04.721806 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:04.909434 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.909277 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" event={"ID":"c412a2301a21d6df633ce7962cca5842","Type":"ContainerStarted","Data":"40fcc61229862496776b892ead9e1a283074f7912f86d5e4f2b42194d290da50"} Sep 15 16:27:04.914373 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.914346 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-crws5" event={"ID":"c86ff085-8efe-41aa-8638-f4b36bf5d910","Type":"ContainerStarted","Data":"eefbac29cf22ba7ca6b59a6a0eaf354f281bf9a2f0e8e80049cdad2834c06d8e"} Sep 15 16:27:04.933610 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.933150 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vj8vc" event={"ID":"1f22ba54-c674-4533-a54a-7e7315f5562b","Type":"ContainerStarted","Data":"02d4a6a094ac420976a6ef80abb6bacfce4650dd9cb69c9b2a7db01bd072cb99"} Sep 15 16:27:04.939173 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.939008 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"74bd402fcc68ab5b2ab1149a4f1d9b0bdd788728b05364db73f30a456ff92f43"} Sep 15 16:27:04.944865 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.944625 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-196.ec2.internal" podStartSLOduration=19.944603099 podStartE2EDuration="19.944603099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:04.944205933 +0000 UTC m=+21.713487762" watchObservedRunningTime="2026-09-15 16:27:04.944603099 +0000 UTC m=+21.713884939" Sep 15 16:27:04.970377 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.970207 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-crws5" podStartSLOduration=1.575724558 podStartE2EDuration="20.970188596s" totalImagesPullingTime="19.394464038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:04.969985977 +0000 UTC m=+21.739267806" watchObservedRunningTime="2026-09-15 16:27:04.970188596 +0000 UTC m=+21.739470423" Sep 15 16:27:04.989690 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:04.989593 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vj8vc" podStartSLOduration=1.5626164120000001 podStartE2EDuration="20.989571692s" totalImagesPullingTime="19.42695528s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:04.988844028 +0000 UTC m=+21.758125854" watchObservedRunningTime="2026-09-15 16:27:04.989571692 +0000 UTC m=+21.758853519" Sep 15 16:27:05.722593 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.722555 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:05.722830 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:05.722715 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:05.944192 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.944105 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"04153c46a953e69c6ee435d9cce12df86b4e6764ef97161b634cf9981fcfa6ea"} Sep 15 16:27:05.944192 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.944176 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"4b841c907fef7b7b6cb8fbeedc9ce05dc6568d63a2eea743adc2688e5dd1aaae"} Sep 15 16:27:05.944965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.944222 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"2f191e0872a03e8d6b0063e97198ce56afd88487f084a7870a2805640851068c"} Sep 15 16:27:05.944965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.944262 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"578a8cc6a450eff07e70af1978757aa5efb234db0faee2a9a389c4c585b7b621"} Sep 15 16:27:05.944965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.944291 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"dae861c32e57d47bdd8042c42e99a08022e06879254544e07c7f7b082b9213c9"} Sep 15 16:27:05.945580 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.945559 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jsdtg" event={"ID":"0ed7aa9b-df82-4e1a-b35c-89087c2ced64","Type":"ContainerStarted","Data":"153a7f5b07420975e76b9bcc82145bfe6f216dbf992cb75cc2e8bf50a101c616"} Sep 15 16:27:05.946960 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.946927 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdxnf" event={"ID":"31f363f1-7083-44b7-b0c9-8ee80b2ded87","Type":"ContainerStarted","Data":"54c5eba12a15e61407bd4da6289866a0008f780dd6f69206df6a7eccef1da426"} Sep 15 16:27:05.948355 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.948332 2558 generic.go:356] "Generic (PLEG): container finished" podID="ffe9df7a3782f64b64b388867dcd4e48" containerID="c27242e5b707a4e6b7511be7f1957c4f4397eaf7794bdda668f59814ab404e69" exitCode=0 Sep 15 16:27:05.948445 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.948405 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" event={"ID":"ffe9df7a3782f64b64b388867dcd4e48","Type":"ContainerDied","Data":"c27242e5b707a4e6b7511be7f1957c4f4397eaf7794bdda668f59814ab404e69"} Sep 15 16:27:05.950046 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.950011 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" event={"ID":"da83f919-25b3-4d35-872b-d9634c7a1cc3","Type":"ContainerStarted","Data":"11ab56af503305df99e00cd371ee09a149e7372b9b3b8e91f3e4aa459b15bb93"} Sep 15 16:27:05.951423 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.951392 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ct27n" event={"ID":"0cd19c25-3c74-418d-bab8-dc432ccc1e88","Type":"ContainerStarted","Data":"69598b9b391b96e04693a1c1970e66f62182d2a91f63d64feae3918afef5a129"} Sep 15 16:27:05.952828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.952808 2558 generic.go:356] "Generic (PLEG): container finished" podID="04db1029-90cb-4a75-a9c6-c0d1567cb33d" containerID="c93c32b68a2473305ed23b7875a0e567cf05842823c983d56da79045063f0c27" exitCode=0 Sep 15 16:27:05.952894 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.952834 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerDied","Data":"c93c32b68a2473305ed23b7875a0e567cf05842823c983d56da79045063f0c27"} Sep 15 16:27:05.954440 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.954417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9ctfv" event={"ID":"480f2cdd-804a-4d7f-b865-3ddd4da269c7","Type":"ContainerStarted","Data":"f2678493cda7c8148d19ea6400e707072ad26ed2107d2290aea24b05b1ab556b"} Sep 15 16:27:05.961848 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.960446 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jsdtg" podStartSLOduration=2.624026217 podStartE2EDuration="21.960429875s" totalImagesPullingTime="19.336403658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:05.958926254 +0000 UTC m=+22.728208080" watchObservedRunningTime="2026-09-15 16:27:05.960429875 +0000 UTC m=+22.729711706" Sep 15 16:27:05.974076 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:05.974043 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ct27n" podStartSLOduration=1.716948034 podStartE2EDuration="20.974027928s" totalImagesPullingTime="19.257079894s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:05.97354884 +0000 UTC m=+22.742830668" watchObservedRunningTime="2026-09-15 16:27:05.974027928 +0000 UTC m=+22.743309756" Sep 15 16:27:06.007670 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.007465 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gdxnf" podStartSLOduration=2.654627826 podStartE2EDuration="22.007445577s" totalImagesPullingTime="19.352817751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:05.990515301 +0000 UTC m=+22.759797132" watchObservedRunningTime="2026-09-15 16:27:06.007445577 +0000 UTC m=+22.776727409" Sep 15 16:27:06.007925 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.007895 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9ctfv" podStartSLOduration=2.6873312609999997 podStartE2EDuration="22.007882347s" totalImagesPullingTime="19.320551086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:06.006971769 +0000 UTC m=+22.776253597" watchObservedRunningTime="2026-09-15 16:27:06.007882347 +0000 UTC m=+22.777164174" Sep 15 16:27:06.578122 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.578096 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 16:27:06.660709 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.660603 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T16:27:06.578116026Z","UUID":"1f1ceea4-3e18-4bac-a2db-2af429c4d6b9","Handler":null,"Name":"","Endpoint":""} Sep 15 16:27:06.662379 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.662358 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 16:27:06.662379 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.662388 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 16:27:06.721791 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.721714 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:06.721925 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.721719 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:06.721925 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:06.721836 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:06.721925 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:06.721892 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:06.958343 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.958306 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" event={"ID":"ffe9df7a3782f64b64b388867dcd4e48","Type":"ContainerStarted","Data":"a1007d97220f7152c44b826df46da32f82ea8ef8b23039d17c0ea6b8e85fad57"} Sep 15 16:27:06.960106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.960076 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" event={"ID":"da83f919-25b3-4d35-872b-d9634c7a1cc3","Type":"ContainerStarted","Data":"34e6e80f2a17fce8662df73e3d09878c8e2c4fd900a27bac991c53d1ce50ec38"} Sep 15 16:27:06.975496 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:06.975430 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-196.ec2.internal" podStartSLOduration=21.975416285 podStartE2EDuration="21.975416285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:06.974971657 +0000 UTC m=+23.744253483" watchObservedRunningTime="2026-09-15 16:27:06.975416285 +0000 UTC m=+23.744698111" Sep 15 16:27:07.721854 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:07.721823 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:07.722001 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:07.721945 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:07.965504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:07.965470 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"afd33b95f6e8713b6cd858803050733aa0a0d9ad7d5e929fb46fac8b3b1d4b3f"} Sep 15 16:27:07.967515 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:07.967493 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" event={"ID":"da83f919-25b3-4d35-872b-d9634c7a1cc3","Type":"ContainerStarted","Data":"8eed769be522d51f158417fb3da6250dacdf97d22b675dc70737a88a2c74dcfe"} Sep 15 16:27:07.983539 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:07.983465 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rxmtg" podStartSLOduration=1.07816099 podStartE2EDuration="22.983380199s" totalImagesPullingTime="21.905219209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:07.983271762 +0000 UTC m=+24.752553589" watchObservedRunningTime="2026-09-15 16:27:07.983380199 +0000 UTC m=+24.752662027" Sep 15 16:27:08.721807 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:08.721776 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:08.722024 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:08.721776 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:08.722024 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:08.721887 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:08.722024 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:08.721970 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:09.722318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:09.722277 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:09.722779 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:09.722419 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:10.480621 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:10.480570 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:27:10.481209 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:10.481184 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:27:10.722160 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:10.722123 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:10.722304 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:10.722121 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:10.722304 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:10.722251 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:10.722623 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:10.722313 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:10.974757 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:10.974710 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:27:10.975551 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:10.975254 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gdxnf" Sep 15 16:27:11.721924 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:11.721756 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:11.722085 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:11.722016 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:11.978202 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:11.978125 2558 generic.go:356] "Generic (PLEG): container finished" podID="04db1029-90cb-4a75-a9c6-c0d1567cb33d" containerID="18ca38cdb4e69d3a95ecf2b79e8250ea75dc9d664f79a9facc66da7e5a96dd77" exitCode=0 Sep 15 16:27:11.978849 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:11.978203 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerDied","Data":"18ca38cdb4e69d3a95ecf2b79e8250ea75dc9d664f79a9facc66da7e5a96dd77"} Sep 15 16:27:11.982279 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:11.982255 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" event={"ID":"f36bf4e7-e893-4d81-8285-66f26193a29a","Type":"ContainerStarted","Data":"fed489c19876681c1ac48aad15231c91ac05db3ad00c43eb3a0641f2cf310e53"} Sep 15 16:27:12.029201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.029156 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" podStartSLOduration=8.198496976 podStartE2EDuration="28.02914229s" totalImagesPullingTime="19.830645314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:12.027502186 +0000 UTC m=+28.796784012" watchObservedRunningTime="2026-09-15 16:27:12.02914229 +0000 UTC m=+28.798424117" Sep 15 16:27:12.722388 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.722361 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:12.722512 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:12.722461 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:12.722512 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.722488 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:12.722573 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:12.722540 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:12.986955 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.986928 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerStarted","Data":"d5535ea82761526bf5432dcd59150cf09ee11100afc741fec2880f0430e0576e"} Sep 15 16:27:12.987558 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.987538 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:27:12.987641 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.987605 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:27:12.987691 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:12.987647 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:27:13.001906 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.001884 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:27:13.002091 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.002069 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:27:13.107305 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.107275 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82z9t"] Sep 15 16:27:13.107441 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.107367 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:13.107500 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:13.107480 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:13.118498 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.118466 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vgmpk"] Sep 15 16:27:13.118612 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.118541 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:13.118667 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:13.118624 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:13.124383 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.124360 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wlfzm"] Sep 15 16:27:13.124477 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.124458 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:13.124581 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:13.124557 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:13.991060 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.991015 2558 generic.go:356] "Generic (PLEG): container finished" podID="04db1029-90cb-4a75-a9c6-c0d1567cb33d" containerID="d5535ea82761526bf5432dcd59150cf09ee11100afc741fec2880f0430e0576e" exitCode=0 Sep 15 16:27:13.991489 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:13.991096 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerDied","Data":"d5535ea82761526bf5432dcd59150cf09ee11100afc741fec2880f0430e0576e"} Sep 15 16:27:14.721781 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:14.721603 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:14.721953 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:14.721603 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:14.721953 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:14.721864 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:14.721953 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:14.721638 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:14.721953 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:14.721928 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:14.722242 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:14.722007 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:15.996762 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:15.996704 2558 generic.go:356] "Generic (PLEG): container finished" podID="04db1029-90cb-4a75-a9c6-c0d1567cb33d" containerID="56754527ec980d9b5fb04c08622f2e62a114acc48f3f031fb6d8e873268dfedf" exitCode=0 Sep 15 16:27:15.997219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:15.996772 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerDied","Data":"56754527ec980d9b5fb04c08622f2e62a114acc48f3f031fb6d8e873268dfedf"} Sep 15 16:27:16.458188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:16.458110 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:16.458188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:16.458172 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:16.458378 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.458255 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:27:16.458378 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.458313 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs podName:1a2f1911-ce77-48aa-b3b8-c2199a13c65c nodeName:}" failed. No retries permitted until 2026-09-15 16:27:48.458299673 +0000 UTC m=+65.227581478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs") pod "network-metrics-daemon-wlfzm" (UID: "1a2f1911-ce77-48aa-b3b8-c2199a13c65c") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 16:27:16.458378 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.458255 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 16:27:16.458378 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.458369 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret podName:6720930f-c883-40dc-bea0-11fc91a597c9 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:48.458355865 +0000 UTC m=+65.227637685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret") pod "global-pull-secret-syncer-82z9t" (UID: "6720930f-c883-40dc-bea0-11fc91a597c9") : object "kube-system"/"original-pull-secret" not registered Sep 15 16:27:16.558865 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:16.558831 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:16.559006 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.558956 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 16:27:16.559006 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.558970 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 16:27:16.559006 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.558979 2558 projected.go:196] Error preparing data for projected volume kube-api-access-2f9g7 for pod openshift-network-diagnostics/network-check-target-vgmpk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:27:16.559109 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.559026 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7 podName:f1ac928e-2633-4ad0-9ab6-7c12f05c8610 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:48.55900931 +0000 UTC m=+65.328291118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2f9g7" (UniqueName: "kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7") pod "network-check-target-vgmpk" (UID: "f1ac928e-2633-4ad0-9ab6-7c12f05c8610") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 16:27:16.722077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:16.722054 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:16.722205 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:16.722054 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:16.722205 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.722146 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wlfzm" podUID="1a2f1911-ce77-48aa-b3b8-c2199a13c65c" Sep 15 16:27:16.722304 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.722210 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vgmpk" podUID="f1ac928e-2633-4ad0-9ab6-7c12f05c8610" Sep 15 16:27:16.722304 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:16.722054 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:16.722400 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:16.722319 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82z9t" podUID="6720930f-c883-40dc-bea0-11fc91a597c9" Sep 15 16:27:17.481554 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.481485 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-196.ec2.internal" event="NodeReady" Sep 15 16:27:17.481891 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.481600 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 16:27:17.516904 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.516878 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56644b7889-2zl7k"] Sep 15 16:27:17.535677 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.535651 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w"] Sep 15 16:27:17.535827 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.535807 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.540053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.540006 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 15 16:27:17.540216 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.540159 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 15 16:27:17.540304 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.540221 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7jbp8\"" Sep 15 16:27:17.541365 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.541344 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 15 16:27:17.548971 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.548949 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 15 16:27:17.550375 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.550354 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w"] Sep 15 16:27:17.550686 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.550624 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.553327 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.553308 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 15 16:27:17.553327 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.553331 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.553471 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.553342 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 15 16:27:17.553471 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.553352 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-85k7q\"" Sep 15 16:27:17.554164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.554150 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.567074 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.567055 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56644b7889-2zl7k"] Sep 15 16:27:17.567188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.567082 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w"] Sep 15 16:27:17.567188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.567100 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bfl42"] Sep 15 16:27:17.567296 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.567214 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.569832 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.569813 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 15 16:27:17.571192 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.571171 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.571292 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.571199 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 15 16:27:17.571349 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.571181 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.571495 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.571470 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lm954\"" Sep 15 16:27:17.583617 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.583593 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w"] Sep 15 16:27:17.583754 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.583626 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq"] Sep 15 16:27:17.583754 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.583714 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.586015 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.585999 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 15 16:27:17.586376 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.586349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 15 16:27:17.586474 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.586401 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.586946 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.586930 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-97qzc\"" Sep 15 16:27:17.587193 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.587174 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.591438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.591420 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 15 16:27:17.604279 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.604260 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth"] Sep 15 16:27:17.604360 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.604351 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" Sep 15 16:27:17.606543 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.606527 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.606884 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.606865 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.606977 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.606910 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dl7f5\"" Sep 15 16:27:17.616892 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.616875 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-795444bb76-td5xf"] Sep 15 16:27:17.616987 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.616974 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:17.619245 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.619201 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 15 16:27:17.619245 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.619210 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mjn4h\"" Sep 15 16:27:17.619245 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.619206 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.619454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.619440 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.632465 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.631342 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw"] Sep 15 16:27:17.632549 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.632509 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.635370 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.635354 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 15 16:27:17.635471 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.635420 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6cbkl\"" Sep 15 16:27:17.635802 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.635785 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 15 16:27:17.635863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.635823 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 15 16:27:17.635916 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.635862 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 15 16:27:17.636062 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.636050 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.636135 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.636121 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.652689 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.652669 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-n5jpm"] Sep 15 16:27:17.652810 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.652797 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.655231 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.655209 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.655342 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.655323 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4mc68\"" Sep 15 16:27:17.655342 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.655342 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.655477 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.655363 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 15 16:27:17.655477 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.655426 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 15 16:27:17.666360 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666342 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndvd\" (UniqueName: \"kubernetes.io/projected/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-kube-api-access-qndvd\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.666438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666367 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.666438 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666399 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-bound-sa-token\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666501 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666450 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.666501 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666470 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5h4\" (UniqueName: \"kubernetes.io/projected/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-kube-api-access-dr5h4\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.666501 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666487 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-image-registry-private-configuration\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666589 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666512 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8d9\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-kube-api-access-xp8d9\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666589 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666527 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-serving-cert\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.666589 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666540 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-config\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.666589 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666559 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-config\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.666589 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666581 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-serving-cert\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.666748 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666599 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vh5\" (UniqueName: \"kubernetes.io/projected/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-kube-api-access-k9vh5\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.666748 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666614 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-trusted-ca\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.666748 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666628 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666748 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666643 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-installation-pull-secrets\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666748 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666695 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffc94ac-2e9b-41cd-82be-a05528c51c25-ca-trust-extracted\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666895 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666717 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-certificates\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.666895 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.666767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-trusted-ca\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.670522 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.670440 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zjwff"] Sep 15 16:27:17.670665 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.670648 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.673389 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.673372 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 15 16:27:17.673538 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.673519 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.673600 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.673551 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 15 16:27:17.674812 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.674797 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-27v6r\"" Sep 15 16:27:17.675014 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.674995 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.679370 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.679353 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 15 16:27:17.683564 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683545 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bfl42"] Sep 15 16:27:17.683646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683575 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth"] Sep 15 16:27:17.683646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683594 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zjwff"] Sep 15 16:27:17.683646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683608 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw"] Sep 15 16:27:17.683646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683619 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-n5jpm"] Sep 15 16:27:17.683646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683637 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k68km"] Sep 15 16:27:17.683873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.683671 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.686770 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.686753 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 16:27:17.686853 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.686811 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xg4rz\"" Sep 15 16:27:17.686853 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.686848 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 16:27:17.702180 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.702162 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c"] Sep 15 16:27:17.702283 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.702269 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:17.704342 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.704326 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-55j2f\"" Sep 15 16:27:17.704417 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.704326 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.704603 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.704588 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.704703 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.704689 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 16:27:17.714371 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.714354 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq"] Sep 15 16:27:17.714450 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.714381 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-795444bb76-td5xf"] Sep 15 16:27:17.714450 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.714396 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c"] Sep 15 16:27:17.714450 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.714411 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k68km"] Sep 15 16:27:17.714450 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.714437 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt"] Sep 15 16:27:17.714588 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.714574 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" Sep 15 16:27:17.716945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.716928 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 16:27:17.717036 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.716954 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 16:27:17.717036 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.717005 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-kd97h\"" Sep 15 16:27:17.730832 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.730813 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:17.733068 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.733012 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 15 16:27:17.733166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.733076 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 15 16:27:17.733166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.733080 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mvfl2\"" Sep 15 16:27:17.734598 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.734580 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt"] Sep 15 16:27:17.767768 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.767744 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-serving-cert\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.767883 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.767777 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-image-registry-private-configuration\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.767883 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.767798 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-default-certificate\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.767883 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.767815 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.768011 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.767986 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-serving-cert\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.768054 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768030 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-installation-pull-secrets\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.768091 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768076 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-certificates\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.768128 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768103 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:17.768178 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768125 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68decd37-fefc-41d4-8a7b-ab6b40e1096b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.768178 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768151 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndvd\" (UniqueName: \"kubernetes.io/projected/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-kube-api-access-qndvd\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.768282 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768226 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5c1b79-6bb9-4718-adde-6280b2032021-serving-cert\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.768336 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768284 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9341790-00f2-4daf-884c-6c0157849da8-tmp-dir\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.768336 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768321 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.768451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768364 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1c5c1b79-6bb9-4718-adde-6280b2032021-snapshots\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.768451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768392 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68decd37-fefc-41d4-8a7b-ab6b40e1096b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.768451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768414 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9341790-00f2-4daf-884c-6c0157849da8-config-volume\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.768451 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.768419 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:17.768637 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768462 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-bound-sa-token\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.768637 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.768538 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls podName:6c5ce5cb-2b43-4718-99ce-22bc47aa26bf nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.268517966 +0000 UTC m=+35.037799786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rgx5w" (UID: "6c5ce5cb-2b43-4718-99ce-22bc47aa26bf") : secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:17.768761 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768657 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-certificates\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.768761 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768710 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8d9\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-kube-api-access-xp8d9\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.768863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-config\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.768863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.768798 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7hl\" (UniqueName: \"kubernetes.io/projected/12c4db27-9234-4d74-8b9f-cd83e1a835f0-kube-api-access-fx7hl\") pod \"volume-data-source-validator-86695b7cc7-smmkq\" (UID: \"12c4db27-9234-4d74-8b9f-cd83e1a835f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" Sep 15 16:27:17.769102 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769009 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.769102 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769074 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5c1b79-6bb9-4718-adde-6280b2032021-tmp\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.769233 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769104 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5h4\" (UniqueName: \"kubernetes.io/projected/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-kube-api-access-dr5h4\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.769233 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769126 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.769233 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-config\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.769403 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769237 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.769403 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769314 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-config\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.769403 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769388 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vh5\" (UniqueName: \"kubernetes.io/projected/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-kube-api-access-k9vh5\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.769560 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769414 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-trusted-ca\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.769560 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769446 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.769560 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769473 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkr76\" (UniqueName: \"kubernetes.io/projected/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-kube-api-access-mkr76\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.769560 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769539 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffc94ac-2e9b-41cd-82be-a05528c51c25-ca-trust-extracted\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-trusted-ca\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769600 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrrg\" (UniqueName: \"kubernetes.io/projected/68decd37-fefc-41d4-8a7b-ab6b40e1096b-kube-api-access-cjrrg\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769630 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22pn\" (UniqueName: \"kubernetes.io/projected/71461fdb-4422-43e4-a0fd-42afb978f646-kube-api-access-f22pn\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769661 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dct\" (UniqueName: \"kubernetes.io/projected/c9341790-00f2-4daf-884c-6c0157849da8-kube-api-access-p6dct\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769698 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlq7h\" (UniqueName: \"kubernetes.io/projected/1c5c1b79-6bb9-4718-adde-6280b2032021-kube-api-access-wlq7h\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769736 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-stats-auth\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769762 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5c1b79-6bb9-4718-adde-6280b2032021-trusted-ca-bundle\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.769787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.769784 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5c1b79-6bb9-4718-adde-6280b2032021-service-ca-bundle\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.770764 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.769964 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 16:27:17.770764 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.769978 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56644b7889-2zl7k: secret "image-registry-tls" not found Sep 15 16:27:17.770764 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.770037 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls podName:5ffc94ac-2e9b-41cd-82be-a05528c51c25 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.270014341 +0000 UTC m=+35.039296147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls") pod "image-registry-56644b7889-2zl7k" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25") : secret "image-registry-tls" not found Sep 15 16:27:17.770764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.770068 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-config\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.770764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.770545 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-trusted-ca\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.770764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.770597 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffc94ac-2e9b-41cd-82be-a05528c51c25-ca-trust-extracted\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.771023 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.770826 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.771143 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.771121 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-trusted-ca\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.772416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.772392 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-serving-cert\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.772496 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.772422 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-serving-cert\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.772496 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.772420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-image-registry-private-configuration\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.772801 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.772782 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-installation-pull-secrets\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.777232 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.777205 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-bound-sa-token\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.777334 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.777307 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndvd\" (UniqueName: \"kubernetes.io/projected/4e672e0c-5a45-4d92-a24e-5bcba01e1adc-kube-api-access-qndvd\") pod \"service-ca-operator-cf7c776b7-tjc6w\" (UID: \"4e672e0c-5a45-4d92-a24e-5bcba01e1adc\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.777531 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.777505 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8d9\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-kube-api-access-xp8d9\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:17.777896 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.777879 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vh5\" (UniqueName: \"kubernetes.io/projected/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-kube-api-access-k9vh5\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:17.777961 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.777894 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5h4\" (UniqueName: \"kubernetes.io/projected/4054a7ce-df3a-4ead-b7c9-032d28a0e1c7-kube-api-access-dr5h4\") pod \"console-operator-696884fb9c-bfl42\" (UID: \"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7\") " pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.859133 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.859102 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" Sep 15 16:27:17.870447 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870405 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1c5c1b79-6bb9-4718-adde-6280b2032021-snapshots\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.870447 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870446 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68decd37-fefc-41d4-8a7b-ab6b40e1096b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.870623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870469 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9341790-00f2-4daf-884c-6c0157849da8-config-volume\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.870623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870499 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7hl\" (UniqueName: \"kubernetes.io/projected/12c4db27-9234-4d74-8b9f-cd83e1a835f0-kube-api-access-fx7hl\") pod \"volume-data-source-validator-86695b7cc7-smmkq\" (UID: \"12c4db27-9234-4d74-8b9f-cd83e1a835f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" Sep 15 16:27:17.870623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870539 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5c1b79-6bb9-4718-adde-6280b2032021-tmp\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.870623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.870623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870592 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870635 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkr76\" (UniqueName: \"kubernetes.io/projected/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-kube-api-access-mkr76\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870669 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrrg\" (UniqueName: \"kubernetes.io/projected/68decd37-fefc-41d4-8a7b-ab6b40e1096b-kube-api-access-cjrrg\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870699 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870748 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfzxw\" (UniqueName: \"kubernetes.io/projected/c84043b1-49cb-4bd9-9dda-ac46fe09b427-kube-api-access-sfzxw\") pod \"ingress-canary-k68km\" (UID: \"c84043b1-49cb-4bd9-9dda-ac46fe09b427\") " pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870777 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22pn\" (UniqueName: \"kubernetes.io/projected/71461fdb-4422-43e4-a0fd-42afb978f646-kube-api-access-f22pn\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dct\" (UniqueName: \"kubernetes.io/projected/c9341790-00f2-4daf-884c-6c0157849da8-kube-api-access-p6dct\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870841 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlq7h\" (UniqueName: \"kubernetes.io/projected/1c5c1b79-6bb9-4718-adde-6280b2032021-kube-api-access-wlq7h\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.870880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870864 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-stats-auth\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.870886 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84043b1-49cb-4bd9-9dda-ac46fe09b427-cert\") pod \"ingress-canary-k68km\" (UID: \"c84043b1-49cb-4bd9-9dda-ac46fe09b427\") " pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.870904 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.370888607 +0000 UTC m=+35.140170431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : configmap references non-existent config key: service-ca.crt Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.870921 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.871006 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.370988314 +0000 UTC m=+35.140270136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : secret "router-metrics-certs-default" not found Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871062 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c9341790-00f2-4daf-884c-6c0157849da8-config-volume\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871116 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5c1b79-6bb9-4718-adde-6280b2032021-trusted-ca-bundle\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871138 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5c1b79-6bb9-4718-adde-6280b2032021-service-ca-bundle\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-default-certificate\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871203 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871226 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9skxd\" (UniqueName: \"kubernetes.io/projected/2f04dcb9-8302-4511-b762-ec05141ca84d-kube-api-access-9skxd\") pod \"network-check-source-6f8b768455-rtb2c\" (UID: \"2f04dcb9-8302-4511-b762-ec05141ca84d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" Sep 15 16:27:17.871259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871257 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871282 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68decd37-fefc-41d4-8a7b-ab6b40e1096b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871299 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5c1b79-6bb9-4718-adde-6280b2032021-serving-cert\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.871333 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.871382 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls podName:c9341790-00f2-4daf-884c-6c0157849da8 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.371368113 +0000 UTC m=+35.140649933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls") pod "dns-default-zjwff" (UID: "c9341790-00f2-4daf-884c-6c0157849da8") : secret "dns-default-metrics-tls" not found Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871646 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9341790-00f2-4daf-884c-6c0157849da8-tmp-dir\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.871766 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871774 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5c1b79-6bb9-4718-adde-6280b2032021-service-ca-bundle\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.871336 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9341790-00f2-4daf-884c-6c0157849da8-tmp-dir\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.871813 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.871810 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls podName:71461fdb-4422-43e4-a0fd-42afb978f646 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.371796256 +0000 UTC m=+35.141078077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wkkth" (UID: "71461fdb-4422-43e4-a0fd-42afb978f646") : secret "samples-operator-tls" not found Sep 15 16:27:17.872308 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.872033 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c5c1b79-6bb9-4718-adde-6280b2032021-trusted-ca-bundle\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.874472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.874016 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-stats-auth\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.874472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.874152 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5c1b79-6bb9-4718-adde-6280b2032021-serving-cert\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.874472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.874273 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-default-certificate\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.877171 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.877146 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68decd37-fefc-41d4-8a7b-ab6b40e1096b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.879876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.879157 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68decd37-fefc-41d4-8a7b-ab6b40e1096b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.881217 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.881145 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c5c1b79-6bb9-4718-adde-6280b2032021-tmp\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.881217 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.881183 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1c5c1b79-6bb9-4718-adde-6280b2032021-snapshots\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.881597 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.881553 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrrg\" (UniqueName: \"kubernetes.io/projected/68decd37-fefc-41d4-8a7b-ab6b40e1096b-kube-api-access-cjrrg\") pod \"kube-storage-version-migrator-operator-6c8f84f897-72tnw\" (UID: \"68decd37-fefc-41d4-8a7b-ab6b40e1096b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.881696 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.881605 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7hl\" (UniqueName: \"kubernetes.io/projected/12c4db27-9234-4d74-8b9f-cd83e1a835f0-kube-api-access-fx7hl\") pod \"volume-data-source-validator-86695b7cc7-smmkq\" (UID: \"12c4db27-9234-4d74-8b9f-cd83e1a835f0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" Sep 15 16:27:17.882423 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.882378 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkr76\" (UniqueName: \"kubernetes.io/projected/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-kube-api-access-mkr76\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:17.884012 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.883280 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlq7h\" (UniqueName: \"kubernetes.io/projected/1c5c1b79-6bb9-4718-adde-6280b2032021-kube-api-access-wlq7h\") pod \"insights-operator-5844b856db-n5jpm\" (UID: \"1c5c1b79-6bb9-4718-adde-6280b2032021\") " pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.884012 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.883705 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dct\" (UniqueName: \"kubernetes.io/projected/c9341790-00f2-4daf-884c-6c0157849da8-kube-api-access-p6dct\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:17.884012 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.883913 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22pn\" (UniqueName: \"kubernetes.io/projected/71461fdb-4422-43e4-a0fd-42afb978f646-kube-api-access-f22pn\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:17.894046 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.893783 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:17.911980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.911951 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" Sep 15 16:27:17.962835 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.961970 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" Sep 15 16:27:17.972795 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.972681 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9skxd\" (UniqueName: \"kubernetes.io/projected/2f04dcb9-8302-4511-b762-ec05141ca84d-kube-api-access-9skxd\") pod \"network-check-source-6f8b768455-rtb2c\" (UID: \"2f04dcb9-8302-4511-b762-ec05141ca84d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.973347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.973476 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.973501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfzxw\" (UniqueName: \"kubernetes.io/projected/c84043b1-49cb-4bd9-9dda-ac46fe09b427-kube-api-access-sfzxw\") pod \"ingress-canary-k68km\" (UID: \"c84043b1-49cb-4bd9-9dda-ac46fe09b427\") " pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.973538 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84043b1-49cb-4bd9-9dda-ac46fe09b427-cert\") pod \"ingress-canary-k68km\" (UID: \"c84043b1-49cb-4bd9-9dda-ac46fe09b427\") " pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.974488 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:17.974582 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert podName:9a8b302b-5c04-4f11-b592-2cb4a1adbfa6 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:18.474560226 +0000 UTC m=+35.243842055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6cfrt" (UID: "9a8b302b-5c04-4f11-b592-2cb4a1adbfa6") : secret "networking-console-plugin-cert" not found Sep 15 16:27:17.975166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.974885 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:17.980886 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.980858 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-n5jpm" Sep 15 16:27:17.981769 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.981289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c84043b1-49cb-4bd9-9dda-ac46fe09b427-cert\") pod \"ingress-canary-k68km\" (UID: \"c84043b1-49cb-4bd9-9dda-ac46fe09b427\") " pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:17.988390 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.988347 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9skxd\" (UniqueName: \"kubernetes.io/projected/2f04dcb9-8302-4511-b762-ec05141ca84d-kube-api-access-9skxd\") pod \"network-check-source-6f8b768455-rtb2c\" (UID: \"2f04dcb9-8302-4511-b762-ec05141ca84d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" Sep 15 16:27:17.988390 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:17.988351 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfzxw\" (UniqueName: \"kubernetes.io/projected/c84043b1-49cb-4bd9-9dda-ac46fe09b427-kube-api-access-sfzxw\") pod \"ingress-canary-k68km\" (UID: \"c84043b1-49cb-4bd9-9dda-ac46fe09b427\") " pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:18.009716 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.009673 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k68km" Sep 15 16:27:18.022811 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.022779 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" Sep 15 16:27:18.172655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.172623 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w"] Sep 15 16:27:18.175500 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.175471 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e672e0c_5a45_4d92_a24e_5bcba01e1adc.slice/crio-44a77b772a4fc47bb94c2f0af6aaaf8d42428d5a6d39f2f81adf7952a6496d94 WatchSource:0}: Error finding container 44a77b772a4fc47bb94c2f0af6aaaf8d42428d5a6d39f2f81adf7952a6496d94: Status 404 returned error can't find the container with id 44a77b772a4fc47bb94c2f0af6aaaf8d42428d5a6d39f2f81adf7952a6496d94 Sep 15 16:27:18.198933 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.198905 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-bfl42"] Sep 15 16:27:18.203249 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.203221 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4054a7ce_df3a_4ead_b7c9_032d28a0e1c7.slice/crio-85a124090359fcef9250d49dcf392567070cba890054c9953e5bd9885fdf0b1e WatchSource:0}: Error finding container 85a124090359fcef9250d49dcf392567070cba890054c9953e5bd9885fdf0b1e: Status 404 returned error can't find the container with id 85a124090359fcef9250d49dcf392567070cba890054c9953e5bd9885fdf0b1e Sep 15 16:27:18.224070 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.224044 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq"] Sep 15 16:27:18.225532 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.225506 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c4db27_9234_4d74_8b9f_cd83e1a835f0.slice/crio-01643a747779f8dc0504109b31d036c9379ea9525beceafb7c766f360f633639 WatchSource:0}: Error finding container 01643a747779f8dc0504109b31d036c9379ea9525beceafb7c766f360f633639: Status 404 returned error can't find the container with id 01643a747779f8dc0504109b31d036c9379ea9525beceafb7c766f360f633639 Sep 15 16:27:18.258882 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.258816 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw"] Sep 15 16:27:18.261803 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.261775 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68decd37_fefc_41d4_8a7b_ab6b40e1096b.slice/crio-90b7da84316af6a5898e35dd223f188e90a5a73d03b3defc30805784384c7326 WatchSource:0}: Error finding container 90b7da84316af6a5898e35dd223f188e90a5a73d03b3defc30805784384c7326: Status 404 returned error can't find the container with id 90b7da84316af6a5898e35dd223f188e90a5a73d03b3defc30805784384c7326 Sep 15 16:27:18.276013 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.275992 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:18.276112 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.276055 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:18.276174 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.276142 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:18.276227 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.276206 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls podName:6c5ce5cb-2b43-4718-99ce-22bc47aa26bf nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.27619167 +0000 UTC m=+36.045473480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rgx5w" (UID: "6c5ce5cb-2b43-4718-99ce-22bc47aa26bf") : secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:18.276287 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.276255 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 16:27:18.276287 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.276273 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56644b7889-2zl7k: secret "image-registry-tls" not found Sep 15 16:27:18.276378 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.276329 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls podName:5ffc94ac-2e9b-41cd-82be-a05528c51c25 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.276311254 +0000 UTC m=+36.045593080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls") pod "image-registry-56644b7889-2zl7k" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25") : secret "image-registry-tls" not found Sep 15 16:27:18.282116 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.282089 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-n5jpm"] Sep 15 16:27:18.285549 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.285526 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5c1b79_6bb9_4718_adde_6280b2032021.slice/crio-9d8b5d2ec6e5f2d9a93be27ef33012691f9dae741fbb1c6c30fed7c0e300828e WatchSource:0}: Error finding container 9d8b5d2ec6e5f2d9a93be27ef33012691f9dae741fbb1c6c30fed7c0e300828e: Status 404 returned error can't find the container with id 9d8b5d2ec6e5f2d9a93be27ef33012691f9dae741fbb1c6c30fed7c0e300828e Sep 15 16:27:18.314639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.314610 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k68km"] Sep 15 16:27:18.317880 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.317854 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84043b1_49cb_4bd9_9dda_ac46fe09b427.slice/crio-1d68e487935c7f1c8e8eff61447cd2f92b5f5c9f68cbb82f1e741d6bcedf848b WatchSource:0}: Error finding container 1d68e487935c7f1c8e8eff61447cd2f92b5f5c9f68cbb82f1e741d6bcedf848b: Status 404 returned error can't find the container with id 1d68e487935c7f1c8e8eff61447cd2f92b5f5c9f68cbb82f1e741d6bcedf848b Sep 15 16:27:18.332652 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.332628 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c"] Sep 15 16:27:18.335371 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:18.335350 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f04dcb9_8302_4511_b762_ec05141ca84d.slice/crio-d2e0f9f82e482e67a5a1283f158207edeaee26cdeac2abcb7823bf88bc00b12c WatchSource:0}: Error finding container d2e0f9f82e482e67a5a1283f158207edeaee26cdeac2abcb7823bf88bc00b12c: Status 404 returned error can't find the container with id d2e0f9f82e482e67a5a1283f158207edeaee26cdeac2abcb7823bf88bc00b12c Sep 15 16:27:18.376639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.376609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:18.376792 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.376668 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:18.376792 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.376756 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:18.376792 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376769 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 16:27:18.376792 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.376783 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:18.377032 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376821 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls podName:c9341790-00f2-4daf-884c-6c0157849da8 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.376802267 +0000 UTC m=+36.146084091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls") pod "dns-default-zjwff" (UID: "c9341790-00f2-4daf-884c-6c0157849da8") : secret "dns-default-metrics-tls" not found Sep 15 16:27:18.377032 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376838 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 16:27:18.377032 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376880 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 16:27:18.377032 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376892 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls podName:71461fdb-4422-43e4-a0fd-42afb978f646 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.376876455 +0000 UTC m=+36.146158265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wkkth" (UID: "71461fdb-4422-43e4-a0fd-42afb978f646") : secret "samples-operator-tls" not found Sep 15 16:27:18.377032 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376916 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.376899771 +0000 UTC m=+36.146181596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : configmap references non-existent config key: service-ca.crt Sep 15 16:27:18.377032 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.376941 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.37693087 +0000 UTC m=+36.146212686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : secret "router-metrics-certs-default" not found Sep 15 16:27:18.478220 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.478194 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:18.478365 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.478347 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 16:27:18.478430 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:18.478421 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert podName:9a8b302b-5c04-4f11-b592-2cb4a1adbfa6 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:19.478401925 +0000 UTC m=+36.247683732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6cfrt" (UID: "9a8b302b-5c04-4f11-b592-2cb4a1adbfa6") : secret "networking-console-plugin-cert" not found Sep 15 16:27:18.722090 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.721972 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:18.722090 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.722009 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:18.722992 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.722752 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:18.725248 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.725034 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 16:27:18.725248 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.725072 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcckz\"" Sep 15 16:27:18.725437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.725349 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9w9qq\"" Sep 15 16:27:18.725562 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:18.725540 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 16:27:19.007222 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.007104 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" event={"ID":"68decd37-fefc-41d4-8a7b-ab6b40e1096b","Type":"ContainerStarted","Data":"90b7da84316af6a5898e35dd223f188e90a5a73d03b3defc30805784384c7326"} Sep 15 16:27:19.008963 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.008927 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-n5jpm" event={"ID":"1c5c1b79-6bb9-4718-adde-6280b2032021","Type":"ContainerStarted","Data":"9d8b5d2ec6e5f2d9a93be27ef33012691f9dae741fbb1c6c30fed7c0e300828e"} Sep 15 16:27:19.011443 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.011417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" event={"ID":"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7","Type":"ContainerStarted","Data":"85a124090359fcef9250d49dcf392567070cba890054c9953e5bd9885fdf0b1e"} Sep 15 16:27:19.013842 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.013817 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k68km" event={"ID":"c84043b1-49cb-4bd9-9dda-ac46fe09b427","Type":"ContainerStarted","Data":"1d68e487935c7f1c8e8eff61447cd2f92b5f5c9f68cbb82f1e741d6bcedf848b"} Sep 15 16:27:19.015622 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.015570 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" event={"ID":"12c4db27-9234-4d74-8b9f-cd83e1a835f0","Type":"ContainerStarted","Data":"01643a747779f8dc0504109b31d036c9379ea9525beceafb7c766f360f633639"} Sep 15 16:27:19.017337 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.017314 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" event={"ID":"4e672e0c-5a45-4d92-a24e-5bcba01e1adc","Type":"ContainerStarted","Data":"44a77b772a4fc47bb94c2f0af6aaaf8d42428d5a6d39f2f81adf7952a6496d94"} Sep 15 16:27:19.019197 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.019155 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" event={"ID":"2f04dcb9-8302-4511-b762-ec05141ca84d","Type":"ContainerStarted","Data":"d2e0f9f82e482e67a5a1283f158207edeaee26cdeac2abcb7823bf88bc00b12c"} Sep 15 16:27:19.287472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.287392 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:19.287629 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.287496 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:19.287696 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.287659 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 16:27:19.287696 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.287676 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56644b7889-2zl7k: secret "image-registry-tls" not found Sep 15 16:27:19.287826 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.287750 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls podName:5ffc94ac-2e9b-41cd-82be-a05528c51c25 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.287717379 +0000 UTC m=+38.056999188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls") pod "image-registry-56644b7889-2zl7k" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25") : secret "image-registry-tls" not found Sep 15 16:27:19.288172 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.288151 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:19.288240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.288214 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls podName:6c5ce5cb-2b43-4718-99ce-22bc47aa26bf nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.28819912 +0000 UTC m=+38.057480927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rgx5w" (UID: "6c5ce5cb-2b43-4718-99ce-22bc47aa26bf") : secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:19.388029 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.387987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:19.388202 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.388086 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:19.388202 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.388115 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:19.388319 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.388226 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:19.388374 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388353 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 16:27:19.388435 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388415 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls podName:c9341790-00f2-4daf-884c-6c0157849da8 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.388397061 +0000 UTC m=+38.157678874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls") pod "dns-default-zjwff" (UID: "c9341790-00f2-4daf-884c-6c0157849da8") : secret "dns-default-metrics-tls" not found Sep 15 16:27:19.388776 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388756 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 16:27:19.388862 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388780 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 16:27:19.388862 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388815 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.388798714 +0000 UTC m=+38.158080540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : secret "router-metrics-certs-default" not found Sep 15 16:27:19.388862 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388857 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls podName:71461fdb-4422-43e4-a0fd-42afb978f646 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.388839366 +0000 UTC m=+38.158121186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wkkth" (UID: "71461fdb-4422-43e4-a0fd-42afb978f646") : secret "samples-operator-tls" not found Sep 15 16:27:19.389031 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.388930 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.388919793 +0000 UTC m=+38.158201604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : configmap references non-existent config key: service-ca.crt Sep 15 16:27:19.489629 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:19.489591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:19.489847 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.489829 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 16:27:19.489935 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:19.489890 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert podName:9a8b302b-5c04-4f11-b592-2cb4a1adbfa6 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:21.489872516 +0000 UTC m=+38.259154334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6cfrt" (UID: "9a8b302b-5c04-4f11-b592-2cb4a1adbfa6") : secret "networking-console-plugin-cert" not found Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.310500 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.310602 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.310657 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.310700 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.310711 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56644b7889-2zl7k: secret "image-registry-tls" not found Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.310757 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls podName:6c5ce5cb-2b43-4718-99ce-22bc47aa26bf nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.310736853 +0000 UTC m=+42.080018673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rgx5w" (UID: "6c5ce5cb-2b43-4718-99ce-22bc47aa26bf") : secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:21.310768 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.310775 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls podName:5ffc94ac-2e9b-41cd-82be-a05528c51c25 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.310767134 +0000 UTC m=+42.080048939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls") pod "image-registry-56644b7889-2zl7k" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25") : secret "image-registry-tls" not found Sep 15 16:27:21.411818 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.411779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:21.411985 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.411861 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:21.411985 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.411882 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:21.411985 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.411929 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 16:27:21.411985 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.411972 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:21.412240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.412007 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls podName:71461fdb-4422-43e4-a0fd-42afb978f646 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.411986148 +0000 UTC m=+42.181267976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wkkth" (UID: "71461fdb-4422-43e4-a0fd-42afb978f646") : secret "samples-operator-tls" not found Sep 15 16:27:21.412240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.412039 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.412022706 +0000 UTC m=+42.181304526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : configmap references non-existent config key: service-ca.crt Sep 15 16:27:21.412240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.412046 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 16:27:21.412240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.412131 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls podName:c9341790-00f2-4daf-884c-6c0157849da8 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.412120876 +0000 UTC m=+42.181402688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls") pod "dns-default-zjwff" (UID: "c9341790-00f2-4daf-884c-6c0157849da8") : secret "dns-default-metrics-tls" not found Sep 15 16:27:21.412240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.412135 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 16:27:21.412240 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.412192 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.412181087 +0000 UTC m=+42.181462891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : secret "router-metrics-certs-default" not found Sep 15 16:27:21.513521 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:21.513473 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:21.513693 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.513652 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 16:27:21.513779 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:21.513752 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert podName:9a8b302b-5c04-4f11-b592-2cb4a1adbfa6 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.513716325 +0000 UTC m=+42.282998141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6cfrt" (UID: "9a8b302b-5c04-4f11-b592-2cb4a1adbfa6") : secret "networking-console-plugin-cert" not found Sep 15 16:27:25.348838 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.348794 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:25.349289 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.348881 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:25.349289 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.348976 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:25.349289 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.349041 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls podName:6c5ce5cb-2b43-4718-99ce-22bc47aa26bf nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.349027388 +0000 UTC m=+50.118309212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rgx5w" (UID: "6c5ce5cb-2b43-4718-99ce-22bc47aa26bf") : secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:25.349289 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.349073 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 16:27:25.349289 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.349098 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56644b7889-2zl7k: secret "image-registry-tls" not found Sep 15 16:27:25.349289 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.349159 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls podName:5ffc94ac-2e9b-41cd-82be-a05528c51c25 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.349141186 +0000 UTC m=+50.118423007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls") pod "image-registry-56644b7889-2zl7k" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25") : secret "image-registry-tls" not found Sep 15 16:27:25.450032 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.449981 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:25.450243 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450135 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 16:27:25.450243 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450200 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls podName:c9341790-00f2-4daf-884c-6c0157849da8 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.450183491 +0000 UTC m=+50.219465302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls") pod "dns-default-zjwff" (UID: "c9341790-00f2-4daf-884c-6c0157849da8") : secret "dns-default-metrics-tls" not found Sep 15 16:27:25.450243 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.450140 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:25.450243 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450225 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 16:27:25.450437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.450298 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:25.450437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.450333 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:25.450437 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450385 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls podName:71461fdb-4422-43e4-a0fd-42afb978f646 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.450334786 +0000 UTC m=+50.219616595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wkkth" (UID: "71461fdb-4422-43e4-a0fd-42afb978f646") : secret "samples-operator-tls" not found Sep 15 16:27:25.450437 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450401 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 16:27:25.450437 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450418 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.450406076 +0000 UTC m=+50.219687883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : configmap references non-existent config key: service-ca.crt Sep 15 16:27:25.450437 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.450438 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.450428009 +0000 UTC m=+50.219709819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : secret "router-metrics-certs-default" not found Sep 15 16:27:25.551144 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:25.551102 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:25.551306 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.551270 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 16:27:25.551361 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:25.551348 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert podName:9a8b302b-5c04-4f11-b592-2cb4a1adbfa6 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.551332542 +0000 UTC m=+50.320614350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6cfrt" (UID: "9a8b302b-5c04-4f11-b592-2cb4a1adbfa6") : secret "networking-console-plugin-cert" not found Sep 15 16:27:29.055437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.055386 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" event={"ID":"2f04dcb9-8302-4511-b762-ec05141ca84d","Type":"ContainerStarted","Data":"5c5db10f9bd798a72a2ab964bd61e657e2c5895b5dded0b741b8d62f158dc8ea"} Sep 15 16:27:29.057268 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.057220 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" event={"ID":"68decd37-fefc-41d4-8a7b-ab6b40e1096b","Type":"ContainerStarted","Data":"595ab4522be1caa69f9f627431bdeac961f950522b64127ef19a974115836192"} Sep 15 16:27:29.060309 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.060286 2558 generic.go:356] "Generic (PLEG): container finished" podID="04db1029-90cb-4a75-a9c6-c0d1567cb33d" containerID="e41acd4605511d2fb787a13709db0e026e7b917c4724f3e4d73d792faa472123" exitCode=0 Sep 15 16:27:29.060430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.060367 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerDied","Data":"e41acd4605511d2fb787a13709db0e026e7b917c4724f3e4d73d792faa472123"} Sep 15 16:27:29.062094 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.062004 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-n5jpm" event={"ID":"1c5c1b79-6bb9-4718-adde-6280b2032021","Type":"ContainerStarted","Data":"3125f6e807425c1252c52ad364f5078b66c94af05616a34518b714f05fdc8d5b"} Sep 15 16:27:29.063693 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.063675 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/0.log" Sep 15 16:27:29.063814 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.063711 2558 generic.go:356] "Generic (PLEG): container finished" podID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" containerID="319719edfc80c6dcd29fdae7c939ecfb305137e458cbfa84226b95cf0dc93093" exitCode=255 Sep 15 16:27:29.063814 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.063759 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" event={"ID":"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7","Type":"ContainerDied","Data":"319719edfc80c6dcd29fdae7c939ecfb305137e458cbfa84226b95cf0dc93093"} Sep 15 16:27:29.063985 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.063970 2558 scope.go:117] "RemoveContainer" containerID="319719edfc80c6dcd29fdae7c939ecfb305137e458cbfa84226b95cf0dc93093" Sep 15 16:27:29.066423 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.065835 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k68km" event={"ID":"c84043b1-49cb-4bd9-9dda-ac46fe09b427","Type":"ContainerStarted","Data":"e23bb56a8fed5b09e24083715ce9dba44ae9d8c7df44d9f244c7b477e4768126"} Sep 15 16:27:29.068036 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.068007 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" event={"ID":"12c4db27-9234-4d74-8b9f-cd83e1a835f0","Type":"ContainerStarted","Data":"fe94e9e575bf5e5209130f09a30c74023a7cddd5e31a6dd05d0aaa6a87f263dd"} Sep 15 16:27:29.069643 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.069624 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" event={"ID":"4e672e0c-5a45-4d92-a24e-5bcba01e1adc","Type":"ContainerStarted","Data":"02b3bc5e7109e9a92ff7d1e230140b45685535edf3a8f5353b896df62e0e370c"} Sep 15 16:27:29.078437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.075745 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-rtb2c" podStartSLOduration=10.424424189 podStartE2EDuration="20.075713766s" totalImagesPullingTime="9.651289577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.072302633 +0000 UTC m=+45.841584461" watchObservedRunningTime="2026-09-15 16:27:29.075713766 +0000 UTC m=+45.844995593" Sep 15 16:27:29.168869 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.168826 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-n5jpm" podStartSLOduration=10.467750226 podStartE2EDuration="20.168806522s" totalImagesPullingTime="9.701056296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.168633695 +0000 UTC m=+45.937915534" watchObservedRunningTime="2026-09-15 16:27:29.168806522 +0000 UTC m=+45.938088348" Sep 15 16:27:29.169487 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.169454 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" podStartSLOduration=10.50363436 podStartE2EDuration="20.169439054s" totalImagesPullingTime="9.665804694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.145314256 +0000 UTC m=+45.914596084" watchObservedRunningTime="2026-09-15 16:27:29.169439054 +0000 UTC m=+45.938720886" Sep 15 16:27:29.188539 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.188084 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k68km" podStartSLOduration=2.190957664 podStartE2EDuration="12.188063433s" totalImagesPullingTime="9.997105769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.186823416 +0000 UTC m=+45.956105246" watchObservedRunningTime="2026-09-15 16:27:29.188063433 +0000 UTC m=+45.957345262" Sep 15 16:27:29.207433 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.207044 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-smmkq" podStartSLOduration=10.446296999 podStartE2EDuration="20.207024318s" totalImagesPullingTime="9.760727319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.206025586 +0000 UTC m=+45.975307395" watchObservedRunningTime="2026-09-15 16:27:29.207024318 +0000 UTC m=+45.976306146" Sep 15 16:27:29.225901 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.225314 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" podStartSLOduration=10.41474258 podStartE2EDuration="20.225294801s" totalImagesPullingTime="9.810552221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.223881548 +0000 UTC m=+45.993163376" watchObservedRunningTime="2026-09-15 16:27:29.225294801 +0000 UTC m=+45.994576629" Sep 15 16:27:29.759927 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.759899 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-br4bh"] Sep 15 16:27:29.779444 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.779406 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.782507 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.782477 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 16:27:29.782710 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.782690 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpbwt\"" Sep 15 16:27:29.782773 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.782758 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 16:27:29.789002 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.788924 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-br4bh"] Sep 15 16:27:29.887976 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.887940 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.887976 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.887978 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.888186 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.888005 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-data-volume\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.888186 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.888087 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvjq\" (UniqueName: \"kubernetes.io/projected/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-api-access-9pvjq\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.888186 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.888143 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-crio-socket\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.988757 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.988716 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-crio-socket\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.988943 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.988900 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.988943 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.988921 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-crio-socket\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.989041 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.988925 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.989041 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.988989 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-data-volume\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.989041 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:29.989005 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 16:27:29.989041 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.989019 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvjq\" (UniqueName: \"kubernetes.io/projected/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-api-access-9pvjq\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.989213 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:29.989049 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.489034314 +0000 UTC m=+47.258316148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : secret "insights-runtime-extractor-tls" not found Sep 15 16:27:29.989213 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:29.989002 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 16:27:29.989213 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:29.989142 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.489123769 +0000 UTC m=+47.258405591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : configmap "kube-rbac-proxy" not found Sep 15 16:27:29.989393 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.989375 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-data-volume\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:29.997791 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:29.997764 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvjq\" (UniqueName: \"kubernetes.io/projected/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-api-access-9pvjq\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:30.076508 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.076437 2558 generic.go:356] "Generic (PLEG): container finished" podID="04db1029-90cb-4a75-a9c6-c0d1567cb33d" containerID="2565ebb363277045ab6d7787dd274e2d3e35f5b958f7cbadb19a361c0b09c27a" exitCode=0 Sep 15 16:27:30.076906 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.076515 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerDied","Data":"2565ebb363277045ab6d7787dd274e2d3e35f5b958f7cbadb19a361c0b09c27a"} Sep 15 16:27:30.078182 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.078167 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/1.log" Sep 15 16:27:30.078581 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.078563 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/0.log" Sep 15 16:27:30.078676 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.078603 2558 generic.go:356] "Generic (PLEG): container finished" podID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" containerID="73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6" exitCode=255 Sep 15 16:27:30.078676 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.078655 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" event={"ID":"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7","Type":"ContainerDied","Data":"73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6"} Sep 15 16:27:30.078791 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.078696 2558 scope.go:117] "RemoveContainer" containerID="319719edfc80c6dcd29fdae7c939ecfb305137e458cbfa84226b95cf0dc93093" Sep 15 16:27:30.079336 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.078978 2558 scope.go:117] "RemoveContainer" containerID="73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6" Sep 15 16:27:30.079336 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:30.079114 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bfl42_openshift-console-operator(4054a7ce-df3a-4ead-b7c9-032d28a0e1c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podUID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" Sep 15 16:27:30.496828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.493509 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:30.496828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:30.493565 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:30.496828 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:30.493801 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 16:27:30.496828 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:30.493879 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:31.493858104 +0000 UTC m=+48.263139926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : secret "insights-runtime-extractor-tls" not found Sep 15 16:27:30.496828 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:30.493893 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 16:27:30.496828 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:30.494014 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:31.493995278 +0000 UTC m=+48.263277102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : configmap "kube-rbac-proxy" not found Sep 15 16:27:31.085406 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.085364 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" event={"ID":"04db1029-90cb-4a75-a9c6-c0d1567cb33d","Type":"ContainerStarted","Data":"d23b751950bcb10e95d0dd88dce936025b49e4884d5f8e2a564e50b33603647b"} Sep 15 16:27:31.086838 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.086820 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/1.log" Sep 15 16:27:31.087136 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.087121 2558 scope.go:117] "RemoveContainer" containerID="73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6" Sep 15 16:27:31.087309 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:31.087290 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bfl42_openshift-console-operator(4054a7ce-df3a-4ead-b7c9-032d28a0e1c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podUID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" Sep 15 16:27:31.116169 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.116121 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cj4xv" podStartSLOduration=8.582827419000001 podStartE2EDuration="47.116104835s" totalImagesPullingTime="38.533277416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:31.115592099 +0000 UTC m=+47.884873927" watchObservedRunningTime="2026-09-15 16:27:31.116104835 +0000 UTC m=+47.885386662" Sep 15 16:27:31.503780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.503746 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:31.503780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.503782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:31.503981 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:31.503873 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 16:27:31.503981 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:31.503910 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 16:27:31.503981 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:31.503950 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.503934845 +0000 UTC m=+50.273216650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : configmap "kube-rbac-proxy" not found Sep 15 16:27:31.503981 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:31.503965 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:33.503958763 +0000 UTC m=+50.273240569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : secret "insights-runtime-extractor-tls" not found Sep 15 16:27:31.525607 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:31.525585 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jsdtg_0ed7aa9b-df82-4e1a-b35c-89087c2ced64/dns-node-resolver/0.log" Sep 15 16:27:32.921359 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:32.921334 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ct27n_0cd19c25-3c74-418d-bab8-dc432ccc1e88/node-ca/0.log" Sep 15 16:27:33.420302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.420270 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:33.420494 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.420332 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:33.420494 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.420425 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 15 16:27:33.420494 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.420437 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56644b7889-2zl7k: secret "image-registry-tls" not found Sep 15 16:27:33.420494 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.420427 2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:33.420494 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.420486 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls podName:5ffc94ac-2e9b-41cd-82be-a05528c51c25 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.420468269 +0000 UTC m=+66.189750089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls") pod "image-registry-56644b7889-2zl7k" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25") : secret "image-registry-tls" not found Sep 15 16:27:33.420692 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.420534 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls podName:6c5ce5cb-2b43-4718-99ce-22bc47aa26bf nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.420518787 +0000 UTC m=+66.189800606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-rgx5w" (UID: "6c5ce5cb-2b43-4718-99ce-22bc47aa26bf") : secret "cluster-monitoring-operator-tls" not found Sep 15 16:27:33.521349 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521307 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521377 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521396 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521436 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521453 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521452 2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521534 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls podName:71461fdb-4422-43e4-a0fd-42afb978f646 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.521515853 +0000 UTC m=+66.290797657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-wkkth" (UID: "71461fdb-4422-43e4-a0fd-42afb978f646") : secret "samples-operator-tls" not found Sep 15 16:27:33.521541 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521536 2558 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521561 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.521541524 +0000 UTC m=+66.290823332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : configmap references non-existent config key: service-ca.crt Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521596 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521596 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521635 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:37.521622494 +0000 UTC m=+54.290904298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : secret "insights-runtime-extractor-tls" not found Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521654 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521658 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs podName:5c23cb37-7ccf-4a86-9b6f-25b4bc588865 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.521650631 +0000 UTC m=+66.290932435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs") pod "router-default-795444bb76-td5xf" (UID: "5c23cb37-7ccf-4a86-9b6f-25b4bc588865") : secret "router-metrics-certs-default" not found Sep 15 16:27:33.521906 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.521713 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls podName:c9341790-00f2-4daf-884c-6c0157849da8 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.521698824 +0000 UTC m=+66.290980633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls") pod "dns-default-zjwff" (UID: "c9341790-00f2-4daf-884c-6c0157849da8") : secret "dns-default-metrics-tls" not found Sep 15 16:27:33.522164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.521967 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:33.522978 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.522960 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k68km_c84043b1-49cb-4bd9-9dda-ac46fe09b427/serve-healthcheck-canary/0.log" Sep 15 16:27:33.622871 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:33.622836 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:33.623043 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.622983 2558 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 15 16:27:33.623084 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:33.623047 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert podName:9a8b302b-5c04-4f11-b592-2cb4a1adbfa6 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.623032611 +0000 UTC m=+66.392314415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-6cfrt" (UID: "9a8b302b-5c04-4f11-b592-2cb4a1adbfa6") : secret "networking-console-plugin-cert" not found Sep 15 16:27:37.557868 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:37.557826 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:37.558404 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:37.557951 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 16:27:37.558404 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:37.558003 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls podName:94cdbf57-fdeb-47f4-a90d-a2e636f0d156 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:45.557990511 +0000 UTC m=+62.327272316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls") pod "insights-runtime-extractor-br4bh" (UID: "94cdbf57-fdeb-47f4-a90d-a2e636f0d156") : secret "insights-runtime-extractor-tls" not found Sep 15 16:27:37.894634 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:37.894546 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:37.894634 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:37.894606 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:37.895002 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:37.894989 2558 scope.go:117] "RemoveContainer" containerID="73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6" Sep 15 16:27:37.895189 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:37.895165 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-bfl42_openshift-console-operator(4054a7ce-df3a-4ead-b7c9-032d28a0e1c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podUID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" Sep 15 16:27:45.003940 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:45.003905 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rwngp" Sep 15 16:27:45.628862 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:45.628821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:45.631075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:45.631044 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/94cdbf57-fdeb-47f4-a90d-a2e636f0d156-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-br4bh\" (UID: \"94cdbf57-fdeb-47f4-a90d-a2e636f0d156\") " pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:45.693657 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:45.693630 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jpbwt\"" Sep 15 16:27:45.701499 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:45.701475 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-br4bh" Sep 15 16:27:45.968564 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:45.968533 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-br4bh"] Sep 15 16:27:45.971511 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:45.971479 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cdbf57_fdeb_47f4_a90d_a2e636f0d156.slice/crio-2028df75d4c063ba55e7a3b5e3490e39b84fd7fbfb5b2a8ade76e6305c5122ac WatchSource:0}: Error finding container 2028df75d4c063ba55e7a3b5e3490e39b84fd7fbfb5b2a8ade76e6305c5122ac: Status 404 returned error can't find the container with id 2028df75d4c063ba55e7a3b5e3490e39b84fd7fbfb5b2a8ade76e6305c5122ac Sep 15 16:27:46.136973 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:46.136937 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-br4bh" event={"ID":"94cdbf57-fdeb-47f4-a90d-a2e636f0d156","Type":"ContainerStarted","Data":"8946c86b5be557a9437f209a52bf9ed7651a01f6b575ce832817309d07066703"} Sep 15 16:27:46.137347 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:46.137019 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-br4bh" event={"ID":"94cdbf57-fdeb-47f4-a90d-a2e636f0d156","Type":"ContainerStarted","Data":"2028df75d4c063ba55e7a3b5e3490e39b84fd7fbfb5b2a8ade76e6305c5122ac"} Sep 15 16:27:47.141681 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:47.141645 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-br4bh" event={"ID":"94cdbf57-fdeb-47f4-a90d-a2e636f0d156","Type":"ContainerStarted","Data":"c28865e350a91d4408cb162489c2e71654772c3adf822e31130e096905439b6f"} Sep 15 16:27:48.146626 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.146594 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-br4bh" event={"ID":"94cdbf57-fdeb-47f4-a90d-a2e636f0d156","Type":"ContainerStarted","Data":"b1fbe37428e3887006c43de849e55f4c4d646ea782c0c836fe9dfa71a0c93691"} Sep 15 16:27:48.167646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.167598 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-br4bh" podStartSLOduration=17.322141041 podStartE2EDuration="19.167582363s" totalImagesPullingTime="1.845441322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:48.166653827 +0000 UTC m=+64.935935655" watchObservedRunningTime="2026-09-15 16:27:48.167582363 +0000 UTC m=+64.936864256" Sep 15 16:27:48.558806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.558773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:48.558977 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.558825 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:48.561980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.561958 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 16:27:48.561980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.561973 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 16:27:48.571386 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.571358 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a2f1911-ce77-48aa-b3b8-c2199a13c65c-metrics-certs\") pod \"network-metrics-daemon-wlfzm\" (UID: \"1a2f1911-ce77-48aa-b3b8-c2199a13c65c\") " pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:48.571477 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.571383 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6720930f-c883-40dc-bea0-11fc91a597c9-original-pull-secret\") pod \"global-pull-secret-syncer-82z9t\" (UID: \"6720930f-c883-40dc-bea0-11fc91a597c9\") " pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:48.659258 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.659228 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:48.661517 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.661493 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9g7\" (UniqueName: \"kubernetes.io/projected/f1ac928e-2633-4ad0-9ab6-7c12f05c8610-kube-api-access-2f9g7\") pod \"network-check-target-vgmpk\" (UID: \"f1ac928e-2633-4ad0-9ab6-7c12f05c8610\") " pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:48.738877 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.738853 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82z9t" Sep 15 16:27:48.748097 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.748074 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9w9qq\"" Sep 15 16:27:48.756355 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.756326 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wlfzm" Sep 15 16:27:48.756619 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.756602 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vcckz\"" Sep 15 16:27:48.765116 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:48.765098 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:49.010651 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.010342 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82z9t"] Sep 15 16:27:49.012805 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:49.012781 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6720930f_c883_40dc_bea0_11fc91a597c9.slice/crio-61cd2e67e0932e1a870f0df2a7bc1e717796f28db73368cae591ebc0125ee257 WatchSource:0}: Error finding container 61cd2e67e0932e1a870f0df2a7bc1e717796f28db73368cae591ebc0125ee257: Status 404 returned error can't find the container with id 61cd2e67e0932e1a870f0df2a7bc1e717796f28db73368cae591ebc0125ee257 Sep 15 16:27:49.046260 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.046215 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wlfzm"] Sep 15 16:27:49.048607 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:49.048584 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2f1911_ce77_48aa_b3b8_c2199a13c65c.slice/crio-7383e0ebb4e9265f0ea0e1ac3c90475bf5d4972f7a1f4267b6bcba3e0882a9be WatchSource:0}: Error finding container 7383e0ebb4e9265f0ea0e1ac3c90475bf5d4972f7a1f4267b6bcba3e0882a9be: Status 404 returned error can't find the container with id 7383e0ebb4e9265f0ea0e1ac3c90475bf5d4972f7a1f4267b6bcba3e0882a9be Sep 15 16:27:49.062173 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.062156 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vgmpk"] Sep 15 16:27:49.064552 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:49.064506 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ac928e_2633_4ad0_9ab6_7c12f05c8610.slice/crio-4c985232046eddc13491a5cef927025739fd20eccb6884d09d91e739d12cee2e WatchSource:0}: Error finding container 4c985232046eddc13491a5cef927025739fd20eccb6884d09d91e739d12cee2e: Status 404 returned error can't find the container with id 4c985232046eddc13491a5cef927025739fd20eccb6884d09d91e739d12cee2e Sep 15 16:27:49.151435 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.151404 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82z9t" event={"ID":"6720930f-c883-40dc-bea0-11fc91a597c9","Type":"ContainerStarted","Data":"61cd2e67e0932e1a870f0df2a7bc1e717796f28db73368cae591ebc0125ee257"} Sep 15 16:27:49.152904 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.152878 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vgmpk" event={"ID":"f1ac928e-2633-4ad0-9ab6-7c12f05c8610","Type":"ContainerStarted","Data":"ab3822d147cbd8b091968c8ecce3b585728520f56d939b542d80321565cf8ae7"} Sep 15 16:27:49.153019 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.152927 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vgmpk" event={"ID":"f1ac928e-2633-4ad0-9ab6-7c12f05c8610","Type":"ContainerStarted","Data":"4c985232046eddc13491a5cef927025739fd20eccb6884d09d91e739d12cee2e"} Sep 15 16:27:49.153019 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.152954 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:27:49.154046 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.154015 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wlfzm" event={"ID":"1a2f1911-ce77-48aa-b3b8-c2199a13c65c","Type":"ContainerStarted","Data":"7383e0ebb4e9265f0ea0e1ac3c90475bf5d4972f7a1f4267b6bcba3e0882a9be"} Sep 15 16:27:49.172520 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.172474 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vgmpk" podStartSLOduration=64.172458657 podStartE2EDuration="1m4.172458657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:49.171305354 +0000 UTC m=+65.940587180" watchObservedRunningTime="2026-09-15 16:27:49.172458657 +0000 UTC m=+65.941740487" Sep 15 16:27:49.467641 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.467614 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:49.467799 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.467759 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:49.470166 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.470143 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"image-registry-56644b7889-2zl7k\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:49.470221 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.470180 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c5ce5cb-2b43-4718-99ce-22bc47aa26bf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-rgx5w\" (UID: \"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:49.568697 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.568655 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:49.568900 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.568706 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:49.568900 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.568745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:49.568900 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.568793 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:49.569421 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.569387 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-service-ca-bundle\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:49.571105 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.571086 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c9341790-00f2-4daf-884c-6c0157849da8-metrics-tls\") pod \"dns-default-zjwff\" (UID: \"c9341790-00f2-4daf-884c-6c0157849da8\") " pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:49.571187 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.571170 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/71461fdb-4422-43e4-a0fd-42afb978f646-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-wkkth\" (UID: \"71461fdb-4422-43e4-a0fd-42afb978f646\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:49.571230 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.571202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c23cb37-7ccf-4a86-9b6f-25b4bc588865-metrics-certs\") pod \"router-default-795444bb76-td5xf\" (UID: \"5c23cb37-7ccf-4a86-9b6f-25b4bc588865\") " pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:49.651024 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.650997 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7jbp8\"" Sep 15 16:27:49.659211 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.659187 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:49.669311 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.669287 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:49.671961 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.671938 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a8b302b-5c04-4f11-b592-2cb4a1adbfa6-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-6cfrt\" (UID: \"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:49.680439 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.680418 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lm954\"" Sep 15 16:27:49.688759 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.688720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" Sep 15 16:27:49.725535 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.724383 2558 scope.go:117] "RemoveContainer" containerID="73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6" Sep 15 16:27:49.730358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.730333 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mjn4h\"" Sep 15 16:27:49.735426 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.735405 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" Sep 15 16:27:49.743486 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.743446 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6cbkl\"" Sep 15 16:27:49.752454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.752321 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:49.794142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.794111 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xg4rz\"" Sep 15 16:27:49.802190 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.802164 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:49.842822 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.842639 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mvfl2\"" Sep 15 16:27:49.849088 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.849020 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" Sep 15 16:27:49.938283 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.938253 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56644b7889-2zl7k"] Sep 15 16:27:49.941173 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:49.941142 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffc94ac_2e9b_41cd_82be_a05528c51c25.slice/crio-e01d53062703c2b7dc99f7ae4733b736c7ac23ee8f09bb236842804fcc51fd3e WatchSource:0}: Error finding container e01d53062703c2b7dc99f7ae4733b736c7ac23ee8f09bb236842804fcc51fd3e: Status 404 returned error can't find the container with id e01d53062703c2b7dc99f7ae4733b736c7ac23ee8f09bb236842804fcc51fd3e Sep 15 16:27:49.974784 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:49.974690 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w"] Sep 15 16:27:50.027324 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.027286 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth"] Sep 15 16:27:50.060489 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.060461 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-795444bb76-td5xf"] Sep 15 16:27:50.096776 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.096745 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zjwff"] Sep 15 16:27:50.125888 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.125860 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt"] Sep 15 16:27:50.157211 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:50.157173 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c23cb37_7ccf_4a86_9b6f_25b4bc588865.slice/crio-8172cfdd5b3e0487d8971189f47218a194f9f366c435245fb0e5e13cb2f508eb WatchSource:0}: Error finding container 8172cfdd5b3e0487d8971189f47218a194f9f366c435245fb0e5e13cb2f508eb: Status 404 returned error can't find the container with id 8172cfdd5b3e0487d8971189f47218a194f9f366c435245fb0e5e13cb2f508eb Sep 15 16:27:50.162484 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:50.162445 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5ce5cb_2b43_4718_99ce_22bc47aa26bf.slice/crio-6a704f7dcb7196e1bc52cf7f5e92405dc319bd491d7c0b61171476df288f81c1 WatchSource:0}: Error finding container 6a704f7dcb7196e1bc52cf7f5e92405dc319bd491d7c0b61171476df288f81c1: Status 404 returned error can't find the container with id 6a704f7dcb7196e1bc52cf7f5e92405dc319bd491d7c0b61171476df288f81c1 Sep 15 16:27:50.162794 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:50.162769 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9341790_00f2_4daf_884c_6c0157849da8.slice/crio-90244391f63a0ea05e9375507aa29818752de4f1e00f791102585c493d26032d WatchSource:0}: Error finding container 90244391f63a0ea05e9375507aa29818752de4f1e00f791102585c493d26032d: Status 404 returned error can't find the container with id 90244391f63a0ea05e9375507aa29818752de4f1e00f791102585c493d26032d Sep 15 16:27:50.164999 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:27:50.164921 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8b302b_5c04_4f11_b592_2cb4a1adbfa6.slice/crio-f07b8070273a80468f5f89c26f43dd30801ce4b130c5363685fec15f52aa853b WatchSource:0}: Error finding container f07b8070273a80468f5f89c26f43dd30801ce4b130c5363685fec15f52aa853b: Status 404 returned error can't find the container with id f07b8070273a80468f5f89c26f43dd30801ce4b130c5363685fec15f52aa853b Sep 15 16:27:50.166830 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.166652 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:27:50.167149 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.167132 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/1.log" Sep 15 16:27:50.167267 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.167170 2558 generic.go:356] "Generic (PLEG): container finished" podID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" containerID="de702b45ddbefeba98888d3daaa183ec0d3cb431a7f18641ecfb32ea762fd993" exitCode=255 Sep 15 16:27:50.167267 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.167237 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" event={"ID":"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7","Type":"ContainerDied","Data":"de702b45ddbefeba98888d3daaa183ec0d3cb431a7f18641ecfb32ea762fd993"} Sep 15 16:27:50.167346 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.167295 2558 scope.go:117] "RemoveContainer" containerID="73f0040a2b58ea89379c27b8d9ed8a8d6137bf07679519cf9d400166d9051dc6" Sep 15 16:27:50.167596 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.167572 2558 scope.go:117] "RemoveContainer" containerID="de702b45ddbefeba98888d3daaa183ec0d3cb431a7f18641ecfb32ea762fd993" Sep 15 16:27:50.167947 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:50.167792 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bfl42_openshift-console-operator(4054a7ce-df3a-4ead-b7c9-032d28a0e1c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podUID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" Sep 15 16:27:50.169106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:50.169013 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" event={"ID":"5ffc94ac-2e9b-41cd-82be-a05528c51c25","Type":"ContainerStarted","Data":"e01d53062703c2b7dc99f7ae4733b736c7ac23ee8f09bb236842804fcc51fd3e"} Sep 15 16:27:51.179230 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.179158 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" event={"ID":"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6","Type":"ContainerStarted","Data":"f07b8070273a80468f5f89c26f43dd30801ce4b130c5363685fec15f52aa853b"} Sep 15 16:27:51.182612 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.182483 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:27:51.185911 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.185148 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wlfzm" event={"ID":"1a2f1911-ce77-48aa-b3b8-c2199a13c65c","Type":"ContainerStarted","Data":"a7a3ed84780cb632de1e201887ccf45e7a838d10a0f577e41196a2f3776a1b7a"} Sep 15 16:27:51.185911 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.185202 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wlfzm" event={"ID":"1a2f1911-ce77-48aa-b3b8-c2199a13c65c","Type":"ContainerStarted","Data":"a1b2e55d8e2068b59949933549759e2c63d6f55aa5dd75b6926ba2b474154ecd"} Sep 15 16:27:51.188877 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.188261 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" event={"ID":"5ffc94ac-2e9b-41cd-82be-a05528c51c25","Type":"ContainerStarted","Data":"53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b"} Sep 15 16:27:51.188877 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.188766 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:27:51.191213 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.191155 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" event={"ID":"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf","Type":"ContainerStarted","Data":"6a704f7dcb7196e1bc52cf7f5e92405dc319bd491d7c0b61171476df288f81c1"} Sep 15 16:27:51.192654 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.192587 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zjwff" event={"ID":"c9341790-00f2-4daf-884c-6c0157849da8","Type":"ContainerStarted","Data":"90244391f63a0ea05e9375507aa29818752de4f1e00f791102585c493d26032d"} Sep 15 16:27:51.195531 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.195036 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-795444bb76-td5xf" event={"ID":"5c23cb37-7ccf-4a86-9b6f-25b4bc588865","Type":"ContainerStarted","Data":"9e9b96607460331982cec62c05eaec8b03ed1e22c4664af0b765f80116bd929b"} Sep 15 16:27:51.195531 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.195076 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-795444bb76-td5xf" event={"ID":"5c23cb37-7ccf-4a86-9b6f-25b4bc588865","Type":"ContainerStarted","Data":"8172cfdd5b3e0487d8971189f47218a194f9f366c435245fb0e5e13cb2f508eb"} Sep 15 16:27:51.197259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.197191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" event={"ID":"71461fdb-4422-43e4-a0fd-42afb978f646","Type":"ContainerStarted","Data":"691eca4c7939bb692204c5e9843e9ebc4110d6dd333d73deb68e1b48f3ff13cb"} Sep 15 16:27:51.202939 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.202607 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wlfzm" podStartSLOduration=65.062926926 podStartE2EDuration="1m6.202589428s" totalImagesPullingTime="1.139662502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:51.201050966 +0000 UTC m=+67.970332794" watchObservedRunningTime="2026-09-15 16:27:51.202589428 +0000 UTC m=+67.971871255" Sep 15 16:27:51.239988 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.239886 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" podStartSLOduration=66.239863015 podStartE2EDuration="1m6.239863015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:51.220514752 +0000 UTC m=+67.989796580" watchObservedRunningTime="2026-09-15 16:27:51.239863015 +0000 UTC m=+68.009144843" Sep 15 16:27:51.241650 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.240543 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-795444bb76-td5xf" podStartSLOduration=42.240526837 podStartE2EDuration="42.240526837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:51.239570763 +0000 UTC m=+68.008852591" watchObservedRunningTime="2026-09-15 16:27:51.240526837 +0000 UTC m=+68.009808667" Sep 15 16:27:51.753705 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.753626 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:51.756420 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:51.756392 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:52.202358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:52.202277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:52.203425 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:52.203398 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-795444bb76-td5xf" Sep 15 16:27:55.215968 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.215879 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" event={"ID":"6c5ce5cb-2b43-4718-99ce-22bc47aa26bf","Type":"ContainerStarted","Data":"41eaa7be007cb01e1a74cef0acc3b30da86f1f2ff27bf2b73d77709d33e8f90c"} Sep 15 16:27:55.218129 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.218103 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zjwff" event={"ID":"c9341790-00f2-4daf-884c-6c0157849da8","Type":"ContainerStarted","Data":"664604a8baaeae6e5bc6199192bc404925161c6c08b03030e126d6f5fbc94207"} Sep 15 16:27:55.218268 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.218154 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zjwff" Sep 15 16:27:55.218268 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.218172 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zjwff" event={"ID":"c9341790-00f2-4daf-884c-6c0157849da8","Type":"ContainerStarted","Data":"a4cc20f852fecf1ed23d6356079d14d864c918f8b62b98dd381749564eb84a93"} Sep 15 16:27:55.219953 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.219931 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" event={"ID":"71461fdb-4422-43e4-a0fd-42afb978f646","Type":"ContainerStarted","Data":"69cb84ea91c9ad93149c82589f98b960bbb381814765da60e056c5bb5cc15cf2"} Sep 15 16:27:55.220055 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.219982 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" event={"ID":"71461fdb-4422-43e4-a0fd-42afb978f646","Type":"ContainerStarted","Data":"2009a793e78ca757e2b61df115def890b3a6a33a8e081ee3d0e0214ca5a815cc"} Sep 15 16:27:55.221503 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.221484 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82z9t" event={"ID":"6720930f-c883-40dc-bea0-11fc91a597c9","Type":"ContainerStarted","Data":"ce4686f94c9db3552cdd36cfea3d9df4c1fcc02fd2e86c9e84f82ff1b373d947"} Sep 15 16:27:55.223030 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.223012 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" event={"ID":"9a8b302b-5c04-4f11-b592-2cb4a1adbfa6","Type":"ContainerStarted","Data":"df7b1e873450c9d5963e9aec1c56bbd1c93cab0db73b812aa3e9fec4a737869a"} Sep 15 16:27:55.237590 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.237553 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-rgx5w" podStartSLOduration=41.718177223 podStartE2EDuration="46.237538833s" totalImagesPullingTime="4.51936161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:55.236837546 +0000 UTC m=+72.006119373" watchObservedRunningTime="2026-09-15 16:27:55.237538833 +0000 UTC m=+72.006820659" Sep 15 16:27:55.259019 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.258980 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zjwff" podStartSLOduration=33.735815711 podStartE2EDuration="38.258949711s" totalImagesPullingTime="4.523134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:55.257760591 +0000 UTC m=+72.027042419" watchObservedRunningTime="2026-09-15 16:27:55.258949711 +0000 UTC m=+72.028231538" Sep 15 16:27:55.279230 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.279195 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-6cfrt" podStartSLOduration=33.760597457 podStartE2EDuration="38.279182699s" totalImagesPullingTime="4.518585242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:55.278772325 +0000 UTC m=+72.048054151" watchObservedRunningTime="2026-09-15 16:27:55.279182699 +0000 UTC m=+72.048464525" Sep 15 16:27:55.305998 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.305954 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-82z9t" podStartSLOduration=64.561627349 podStartE2EDuration="1m10.305941932s" totalImagesPullingTime="5.744314583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:55.304610571 +0000 UTC m=+72.073892408" watchObservedRunningTime="2026-09-15 16:27:55.305941932 +0000 UTC m=+72.075223784" Sep 15 16:27:55.326216 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:55.326175 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-wkkth" podStartSLOduration=41.865314006 podStartE2EDuration="46.326159351s" totalImagesPullingTime="4.460845345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:55.326005915 +0000 UTC m=+72.095287744" watchObservedRunningTime="2026-09-15 16:27:55.326159351 +0000 UTC m=+72.095441180" Sep 15 16:27:57.894839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:57.894800 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:57.895220 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:57.894861 2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:27:57.895220 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:27:57.895176 2558 scope.go:117] "RemoveContainer" containerID="de702b45ddbefeba98888d3daaa183ec0d3cb431a7f18641ecfb32ea762fd993" Sep 15 16:27:57.895352 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:27:57.895333 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bfl42_openshift-console-operator(4054a7ce-df3a-4ead-b7c9-032d28a0e1c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podUID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" Sep 15 16:28:05.229183 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:05.229151 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zjwff" Sep 15 16:28:06.860991 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.860882 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gj8cw"] Sep 15 16:28:06.863900 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.863884 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:06.866564 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.866538 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 16:28:06.866704 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.866608 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 16:28:06.867118 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.867105 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 16:28:06.868307 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.868288 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 16:28:06.873219 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:06.873203 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cj2v6\"" Sep 15 16:28:07.007926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.007893 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-wtmp\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.007926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.007925 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-sys\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.007945 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-root\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.008007 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.008028 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-textfile\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.008087 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.008142 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/323937a7-e84e-4b71-af66-16f98e092a4a-metrics-client-ca\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008377 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.008179 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pg6f\" (UniqueName: \"kubernetes.io/projected/323937a7-e84e-4b71-af66-16f98e092a4a-kube-api-access-7pg6f\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.008377 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.008250 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-tls\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.108946 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.108914 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-wtmp\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.108946 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.108950 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-sys\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.108971 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-root\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109023 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-textfile\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109040 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-sys\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109062 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109081 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-wtmp\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109112 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/323937a7-e84e-4b71-af66-16f98e092a4a-metrics-client-ca\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109147 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109133 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/323937a7-e84e-4b71-af66-16f98e092a4a-root\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109421 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109154 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pg6f\" (UniqueName: \"kubernetes.io/projected/323937a7-e84e-4b71-af66-16f98e092a4a-kube-api-access-7pg6f\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109421 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109197 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-tls\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109493 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109462 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-textfile\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109789 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109766 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/323937a7-e84e-4b71-af66-16f98e092a4a-metrics-client-ca\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.109919 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.109788 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.111488 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.111439 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-tls\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.111566 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.111531 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/323937a7-e84e-4b71-af66-16f98e092a4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.117454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.117430 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pg6f\" (UniqueName: \"kubernetes.io/projected/323937a7-e84e-4b71-af66-16f98e092a4a-kube-api-access-7pg6f\") pod \"node-exporter-gj8cw\" (UID: \"323937a7-e84e-4b71-af66-16f98e092a4a\") " pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.173180 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.173142 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gj8cw" Sep 15 16:28:07.184169 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:28:07.184145 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323937a7_e84e_4b71_af66_16f98e092a4a.slice/crio-2748553ebf76640ec2115abfb5172dffc31af30efdaf3abe96d4dda5fe2e273c WatchSource:0}: Error finding container 2748553ebf76640ec2115abfb5172dffc31af30efdaf3abe96d4dda5fe2e273c: Status 404 returned error can't find the container with id 2748553ebf76640ec2115abfb5172dffc31af30efdaf3abe96d4dda5fe2e273c Sep 15 16:28:07.269211 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.269181 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj8cw" event={"ID":"323937a7-e84e-4b71-af66-16f98e092a4a","Type":"ContainerStarted","Data":"2748553ebf76640ec2115abfb5172dffc31af30efdaf3abe96d4dda5fe2e273c"} Sep 15 16:28:07.869290 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.869260 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:28:07.872592 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.872563 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:07.875286 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875020 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 15 16:28:07.875286 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875025 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t5gqm\"" Sep 15 16:28:07.875286 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875048 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 15 16:28:07.875286 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875055 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 15 16:28:07.875286 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875201 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 15 16:28:07.875604 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875470 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 15 16:28:07.875604 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875505 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 15 16:28:07.875604 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875536 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 15 16:28:07.875782 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875746 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 15 16:28:07.875782 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.875766 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 15 16:28:07.887630 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:07.887609 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:28:08.015295 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015229 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015486 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015324 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-web-config\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015486 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015426 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015486 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015476 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015640 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015551 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015640 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015579 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-config-out\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015640 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015604 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015640 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015627 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015935 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015644 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015935 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015658 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015935 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015673 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pk4\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-kube-api-access-k8pk4\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015935 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015688 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.015935 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.015702 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117040 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117182 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117078 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117182 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117121 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117182 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117138 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-config-out\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117182 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117182 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:28:08.117174 2558 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117182 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117207 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117231 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:28:08.117250 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls podName:c3328151-caad-4540-be88-3493608431dc nodeName:}" failed. No retries permitted until 2026-09-15 16:28:08.617230458 +0000 UTC m=+85.386512263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c3328151-caad-4540-be88-3493608431dc") : secret "alertmanager-main-tls" not found Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117285 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pk4\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-kube-api-access-k8pk4\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117339 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117382 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117437 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117409 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-web-config\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.117964 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.117942 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.118659 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.118085 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.118782 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.118653 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.120764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.120686 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.120861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.120786 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.120861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.120795 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-web-config\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.120981 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.120958 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-config-out\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.121509 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.121209 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.121509 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.121267 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.121838 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.121814 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.122235 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.122212 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.128872 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.128850 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pk4\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-kube-api-access-k8pk4\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.275171 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.275137 2558 generic.go:356] "Generic (PLEG): container finished" podID="323937a7-e84e-4b71-af66-16f98e092a4a" containerID="1d11cffbd083883beeafea97ad8ec29fe26b19dae30cef641361de85fdd66184" exitCode=0 Sep 15 16:28:08.275340 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.275228 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj8cw" event={"ID":"323937a7-e84e-4b71-af66-16f98e092a4a","Type":"ContainerDied","Data":"1d11cffbd083883beeafea97ad8ec29fe26b19dae30cef641361de85fdd66184"} Sep 15 16:28:08.622291 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:08.622258 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:08.622457 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:28:08.622400 2558 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 15 16:28:08.622509 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:28:08.622459 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls podName:c3328151-caad-4540-be88-3493608431dc nodeName:}" failed. No retries permitted until 2026-09-15 16:28:09.622442066 +0000 UTC m=+86.391723872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c3328151-caad-4540-be88-3493608431dc") : secret "alertmanager-main-tls" not found Sep 15 16:28:09.281555 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.281523 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj8cw" event={"ID":"323937a7-e84e-4b71-af66-16f98e092a4a","Type":"ContainerStarted","Data":"fba854bedfec47bdb2778df275473096b11634d13f547b9bb5edaa03fa1e88c3"} Sep 15 16:28:09.281950 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.281574 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gj8cw" event={"ID":"323937a7-e84e-4b71-af66-16f98e092a4a","Type":"ContainerStarted","Data":"48efb0d6b15bea02f83da6eb17e858958a2e579b423fa0c7256fa25faee9d304"} Sep 15 16:28:09.308580 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.308521 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gj8cw" podStartSLOduration=2.353074837 podStartE2EDuration="3.308504864s" totalImagesPullingTime="893.153785ms" totalInitContainerRuntime="62.276242ms" isStatefulPod=true podCreationTimestamp="2026-09-15 16:28:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:09.307592442 +0000 UTC m=+86.076874270" watchObservedRunningTime="2026-09-15 16:28:09.308504864 +0000 UTC m=+86.077786691" Sep 15 16:28:09.631651 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.631561 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:09.634057 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.634026 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:09.663904 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.663876 2558 patch_prober.go:28] interesting pod/image-registry-56644b7889-2zl7k container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 15 16:28:09.664042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.663926 2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" podUID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:28:09.684469 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.684443 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:28:09.722348 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.722314 2558 scope.go:117] "RemoveContainer" containerID="de702b45ddbefeba98888d3daaa183ec0d3cb431a7f18641ecfb32ea762fd993" Sep 15 16:28:09.722564 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:28:09.722540 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-bfl42_openshift-console-operator(4054a7ce-df3a-4ead-b7c9-032d28a0e1c7)\"" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podUID="4054a7ce-df3a-4ead-b7c9-032d28a0e1c7" Sep 15 16:28:09.967006 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:09.966933 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:28:09.970481 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:28:09.970454 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3328151_caad_4540_be88_3493608431dc.slice/crio-cf6c8b3f9f86581a66bc8c53fdb52411f88bdefcd4edf8714cfdfb6c7aa9e4b9 WatchSource:0}: Error finding container cf6c8b3f9f86581a66bc8c53fdb52411f88bdefcd4edf8714cfdfb6c7aa9e4b9: Status 404 returned error can't find the container with id cf6c8b3f9f86581a66bc8c53fdb52411f88bdefcd4edf8714cfdfb6c7aa9e4b9 Sep 15 16:28:10.286505 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:10.286474 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"cf6c8b3f9f86581a66bc8c53fdb52411f88bdefcd4edf8714cfdfb6c7aa9e4b9"} Sep 15 16:28:11.295823 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:11.295788 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a" exitCode=0 Sep 15 16:28:11.296203 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:11.295879 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a"} Sep 15 16:28:13.210655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:13.210582 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:28:13.308564 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:13.308535 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023"} Sep 15 16:28:13.308712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:13.308597 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f"} Sep 15 16:28:13.308712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:13.308622 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab"} Sep 15 16:28:13.308712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:13.308645 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19"} Sep 15 16:28:13.308712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:13.308668 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180"} Sep 15 16:28:14.316475 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:14.316437 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerStarted","Data":"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58"} Sep 15 16:28:14.347603 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:14.347544 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.02747813 podStartE2EDuration="7.347527475s" totalImagesPullingTime="3.302893711s" totalInitContainerRuntime="17.155634ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:14.345749014 +0000 UTC m=+91.115030844" watchObservedRunningTime="2026-09-15 16:28:14.347527475 +0000 UTC m=+91.116809302" Sep 15 16:28:15.704744 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:15.704705 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56644b7889-2zl7k"] Sep 15 16:28:20.171891 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:20.171863 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vgmpk" Sep 15 16:28:21.721993 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:21.721960 2558 scope.go:117] "RemoveContainer" containerID="de702b45ddbefeba98888d3daaa183ec0d3cb431a7f18641ecfb32ea762fd993" Sep 15 16:28:22.349930 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.349904 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:28:22.350098 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.349968 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" event={"ID":"4054a7ce-df3a-4ead-b7c9-032d28a0e1c7","Type":"ContainerStarted","Data":"24677cfbb14262d9c4299f94f1c47cbaeb2a50c354187d206ac6e7ebb5d20174"} Sep 15 16:28:22.350269 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.350246 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:28:22.366781 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.366752 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" Sep 15 16:28:22.370968 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.370928 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-bfl42" podStartSLOduration=63.58774641 podStartE2EDuration="1m13.370911454s" totalImagesPullingTime="9.783165044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:22.370150354 +0000 UTC m=+99.139432181" watchObservedRunningTime="2026-09-15 16:28:22.370911454 +0000 UTC m=+99.140193282" Sep 15 16:28:22.562197 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.562165 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-bgmrq"] Sep 15 16:28:22.565540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.565517 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:22.568052 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.568027 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 15 16:28:22.568163 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.568068 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 15 16:28:22.568357 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.568340 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c8sxr\"" Sep 15 16:28:22.577301 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.577276 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-bgmrq"] Sep 15 16:28:22.630455 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.630371 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcm7b\" (UniqueName: \"kubernetes.io/projected/37fb045f-d9a7-4fd5-a6de-4fc833ed4387-kube-api-access-kcm7b\") pod \"downloads-76f99584f4-bgmrq\" (UID: \"37fb045f-d9a7-4fd5-a6de-4fc833ed4387\") " pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:22.731221 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.731178 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcm7b\" (UniqueName: \"kubernetes.io/projected/37fb045f-d9a7-4fd5-a6de-4fc833ed4387-kube-api-access-kcm7b\") pod \"downloads-76f99584f4-bgmrq\" (UID: \"37fb045f-d9a7-4fd5-a6de-4fc833ed4387\") " pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:22.740253 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.740215 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcm7b\" (UniqueName: \"kubernetes.io/projected/37fb045f-d9a7-4fd5-a6de-4fc833ed4387-kube-api-access-kcm7b\") pod \"downloads-76f99584f4-bgmrq\" (UID: \"37fb045f-d9a7-4fd5-a6de-4fc833ed4387\") " pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:22.875158 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:22.875114 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:23.150606 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:23.150531 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-bgmrq"] Sep 15 16:28:23.153427 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:28:23.153400 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fb045f_d9a7_4fd5_a6de_4fc833ed4387.slice/crio-ee40553564190d0cea99b87988d7d2eec4e29c4d12c71dd61395261ee8361762 WatchSource:0}: Error finding container ee40553564190d0cea99b87988d7d2eec4e29c4d12c71dd61395261ee8361762: Status 404 returned error can't find the container with id ee40553564190d0cea99b87988d7d2eec4e29c4d12c71dd61395261ee8361762 Sep 15 16:28:23.356210 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:23.356172 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-bgmrq" event={"ID":"37fb045f-d9a7-4fd5-a6de-4fc833ed4387","Type":"ContainerStarted","Data":"ee40553564190d0cea99b87988d7d2eec4e29c4d12c71dd61395261ee8361762"} Sep 15 16:28:31.988079 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.988040 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dc75f65f4-lw97r"] Sep 15 16:28:31.993692 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.993665 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:31.997067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.996916 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 15 16:28:31.997067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.996932 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 15 16:28:31.997265 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.997196 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 15 16:28:31.997878 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.997857 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 15 16:28:31.998315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.998298 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2rbtn\"" Sep 15 16:28:31.998315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:31.998306 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 15 16:28:32.004578 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.004554 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc75f65f4-lw97r"] Sep 15 16:28:32.114995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.114960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-service-ca\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.115185 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.115011 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-console-config\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.115185 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.115040 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwngd\" (UniqueName: \"kubernetes.io/projected/57714f1a-b813-40ef-aa25-f0d37473d589-kube-api-access-cwngd\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.115185 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.115068 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-oauth-config\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.115185 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.115122 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-serving-cert\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.115185 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.115151 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-oauth-serving-cert\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.216546 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.216509 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-service-ca\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.216546 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.216552 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-console-config\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.216814 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.216575 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwngd\" (UniqueName: \"kubernetes.io/projected/57714f1a-b813-40ef-aa25-f0d37473d589-kube-api-access-cwngd\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.216814 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.216772 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-oauth-config\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.216939 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.216888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-serving-cert\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.216939 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.216930 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-oauth-serving-cert\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.217318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.217294 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-service-ca\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.217424 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.217393 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-console-config\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.217629 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.217570 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-oauth-serving-cert\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.219619 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.219598 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-serving-cert\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.219883 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.219859 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-oauth-config\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.224656 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.224635 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwngd\" (UniqueName: \"kubernetes.io/projected/57714f1a-b813-40ef-aa25-f0d37473d589-kube-api-access-cwngd\") pod \"console-dc75f65f4-lw97r\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.304851 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.304763 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:32.585757 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:32.585587 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dc75f65f4-lw97r"] Sep 15 16:28:33.405233 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:33.405185 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc75f65f4-lw97r" event={"ID":"57714f1a-b813-40ef-aa25-f0d37473d589","Type":"ContainerStarted","Data":"80b78d37dadafe8a55a14eb6d424df4afc4aefa934655f5023a40c90aaf31e5c"} Sep 15 16:28:39.855842 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.855808 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b468b688c-l6ksj"] Sep 15 16:28:39.858264 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.858241 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.867240 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.867157 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 15 16:28:39.870629 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.870609 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b468b688c-l6ksj"] Sep 15 16:28:39.985091 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985057 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-trusted-ca-bundle\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.985091 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985099 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-oauth-serving-cert\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.985326 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj47n\" (UniqueName: \"kubernetes.io/projected/98a2aad4-981a-4cc4-9808-36a1bd1141e1-kube-api-access-gj47n\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.985326 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985218 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-config\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.985326 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985283 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-serving-cert\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.985446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-oauth-config\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:39.985446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:39.985414 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-service-ca\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086529 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj47n\" (UniqueName: \"kubernetes.io/projected/98a2aad4-981a-4cc4-9808-36a1bd1141e1-kube-api-access-gj47n\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086529 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086539 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-config\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086564 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-serving-cert\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086640 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-oauth-config\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086665 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-service-ca\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086691 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-trusted-ca-bundle\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.086798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.086712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-oauth-serving-cert\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.087660 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.087570 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-config\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.087660 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.087649 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-oauth-serving-cert\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.087886 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.087750 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-service-ca\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.088285 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.088233 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-trusted-ca-bundle\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.090153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.090103 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-serving-cert\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.090238 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.090154 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-oauth-config\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.095825 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.095803 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj47n\" (UniqueName: \"kubernetes.io/projected/98a2aad4-981a-4cc4-9808-36a1bd1141e1-kube-api-access-gj47n\") pod \"console-b468b688c-l6ksj\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.169859 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.169775 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:40.438044 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.437952 2558 generic.go:356] "Generic (PLEG): container finished" podID="1c5c1b79-6bb9-4718-adde-6280b2032021" containerID="3125f6e807425c1252c52ad364f5078b66c94af05616a34518b714f05fdc8d5b" exitCode=0 Sep 15 16:28:40.438044 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.437994 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-n5jpm" event={"ID":"1c5c1b79-6bb9-4718-adde-6280b2032021","Type":"ContainerDied","Data":"3125f6e807425c1252c52ad364f5078b66c94af05616a34518b714f05fdc8d5b"} Sep 15 16:28:40.438441 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.438420 2558 scope.go:117] "RemoveContainer" containerID="3125f6e807425c1252c52ad364f5078b66c94af05616a34518b714f05fdc8d5b" Sep 15 16:28:40.728416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:40.728371 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" podUID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" containerName="registry" containerID="cri-o://53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b" gracePeriod=30 Sep 15 16:28:41.431427 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.431342 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:28:41.443860 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.443808 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-bgmrq" event={"ID":"37fb045f-d9a7-4fd5-a6de-4fc833ed4387","Type":"ContainerStarted","Data":"1e6e99e478ebfb19dfec59466df4ea25b2f80d7054153d5a1271e8fc4a29a3d0"} Sep 15 16:28:41.443976 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.443874 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:41.445384 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.445365 2558 patch_prober.go:28] interesting pod/downloads-76f99584f4-bgmrq container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.21:8080/\": dial tcp 10.132.0.21:8080: connect: connection refused" start-of-body= Sep 15 16:28:41.445529 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.445510 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-bgmrq" podUID="37fb045f-d9a7-4fd5-a6de-4fc833ed4387" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.21:8080/\": dial tcp 10.132.0.21:8080: connect: connection refused" Sep 15 16:28:41.445651 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.445365 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc75f65f4-lw97r" event={"ID":"57714f1a-b813-40ef-aa25-f0d37473d589","Type":"ContainerStarted","Data":"4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69"} Sep 15 16:28:41.447743 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.447690 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-n5jpm" event={"ID":"1c5c1b79-6bb9-4718-adde-6280b2032021","Type":"ContainerStarted","Data":"78be5014bf2154e0bb0561730b5e860ca1d5db50c46903d6e6ae3b10cf09014a"} Sep 15 16:28:41.449131 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.449113 2558 generic.go:356] "Generic (PLEG): container finished" podID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" containerID="53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b" exitCode=0 Sep 15 16:28:41.449218 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.449147 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" event={"ID":"5ffc94ac-2e9b-41cd-82be-a05528c51c25","Type":"ContainerDied","Data":"53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b"} Sep 15 16:28:41.449218 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.449157 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" Sep 15 16:28:41.449218 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.449172 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56644b7889-2zl7k" event={"ID":"5ffc94ac-2e9b-41cd-82be-a05528c51c25","Type":"ContainerDied","Data":"e01d53062703c2b7dc99f7ae4733b736c7ac23ee8f09bb236842804fcc51fd3e"} Sep 15 16:28:41.449218 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.449191 2558 scope.go:117] "RemoveContainer" containerID="53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b" Sep 15 16:28:41.462407 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.462343 2558 scope.go:117] "RemoveContainer" containerID="53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b" Sep 15 16:28:41.462863 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:28:41.462837 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b\": container with ID starting with 53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b not found: ID does not exist" containerID="53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b" Sep 15 16:28:41.462958 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.462869 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b"} err="failed to get container status \"53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b\": rpc error: code = NotFound desc = could not find container \"53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b\": container with ID starting with 53939aabcfcf7545d77e73ae3acc7cd472ded07b3aa0c867ff76ed9947434b6b not found: ID does not exist" Sep 15 16:28:41.478355 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.478305 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dc75f65f4-lw97r" podStartSLOduration=1.780243579 podStartE2EDuration="10.478287846s" totalImagesPullingTime="8.698044267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:41.47752435 +0000 UTC m=+118.246806180" watchObservedRunningTime="2026-09-15 16:28:41.478287846 +0000 UTC m=+118.247569674" Sep 15 16:28:41.499080 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499032 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-installation-pull-secrets\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499215 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499128 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8d9\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-kube-api-access-xp8d9\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499215 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499192 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-image-registry-private-configuration\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499327 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499231 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499327 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499267 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-bound-sa-token\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499327 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499292 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-trusted-ca\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499465 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499338 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-certificates\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.499465 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.499374 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffc94ac-2e9b-41cd-82be-a05528c51c25-ca-trust-extracted\") pod \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\" (UID: \"5ffc94ac-2e9b-41cd-82be-a05528c51c25\") " Sep 15 16:28:41.500057 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.500028 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:28:41.500546 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.500516 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:28:41.502188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.502136 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:28:41.502303 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.502276 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-kube-api-access-xp8d9" (OuterVolumeSpecName: "kube-api-access-xp8d9") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "kube-api-access-xp8d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:28:41.503457 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.503423 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:28:41.503563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.503532 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:28:41.503787 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.503757 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:28:41.509301 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.509248 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffc94ac-2e9b-41cd-82be-a05528c51c25-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5ffc94ac-2e9b-41cd-82be-a05528c51c25" (UID: "5ffc94ac-2e9b-41cd-82be-a05528c51c25"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 16:28:41.520170 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.520121 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-bgmrq" podStartSLOduration=1.338445697 podStartE2EDuration="19.520102162s" totalImagesPullingTime="18.181656465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:41.518782086 +0000 UTC m=+118.288063914" watchObservedRunningTime="2026-09-15 16:28:41.520102162 +0000 UTC m=+118.289383989" Sep 15 16:28:41.566010 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.565818 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b468b688c-l6ksj"] Sep 15 16:28:41.568479 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:28:41.568445 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a2aad4_981a_4cc4_9808_36a1bd1141e1.slice/crio-d097bba3b788efed8671d496568d7be5efee39b1c7f351df9ba19af2d47bf861 WatchSource:0}: Error finding container d097bba3b788efed8671d496568d7be5efee39b1c7f351df9ba19af2d47bf861: Status 404 returned error can't find the container with id d097bba3b788efed8671d496568d7be5efee39b1c7f351df9ba19af2d47bf861 Sep 15 16:28:41.600428 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600402 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8d9\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-kube-api-access-xp8d9\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600433 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-image-registry-private-configuration\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600451 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600465 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ffc94ac-2e9b-41cd-82be-a05528c51c25-bound-sa-token\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600478 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-trusted-ca\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600493 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ffc94ac-2e9b-41cd-82be-a05528c51c25-registry-certificates\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600505 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ffc94ac-2e9b-41cd-82be-a05528c51c25-ca-trust-extracted\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.600540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.600520 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ffc94ac-2e9b-41cd-82be-a05528c51c25-installation-pull-secrets\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:28:41.768295 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.768218 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56644b7889-2zl7k"] Sep 15 16:28:41.775004 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:41.774968 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56644b7889-2zl7k"] Sep 15 16:28:42.305672 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.305630 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:42.305854 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.305701 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:42.311680 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.311652 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:42.457000 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.456959 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b468b688c-l6ksj" event={"ID":"98a2aad4-981a-4cc4-9808-36a1bd1141e1","Type":"ContainerStarted","Data":"ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd"} Sep 15 16:28:42.457443 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.457019 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b468b688c-l6ksj" event={"ID":"98a2aad4-981a-4cc4-9808-36a1bd1141e1","Type":"ContainerStarted","Data":"d097bba3b788efed8671d496568d7be5efee39b1c7f351df9ba19af2d47bf861"} Sep 15 16:28:42.462720 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.462692 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:28:42.468991 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.468967 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-bgmrq" Sep 15 16:28:42.482319 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:42.482272 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b468b688c-l6ksj" podStartSLOduration=3.482252839 podStartE2EDuration="3.482252839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:42.481421504 +0000 UTC m=+119.250703357" watchObservedRunningTime="2026-09-15 16:28:42.482252839 +0000 UTC m=+119.251534668" Sep 15 16:28:43.727962 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:43.727931 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" path="/var/lib/kubelet/pods/5ffc94ac-2e9b-41cd-82be-a05528c51c25/volumes" Sep 15 16:28:50.169965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:50.169931 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:50.170586 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:50.170563 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:50.175963 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:50.175925 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:50.496782 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:50.496753 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:28:50.542155 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:50.542112 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc75f65f4-lw97r"] Sep 15 16:28:59.528369 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:59.528337 2558 generic.go:356] "Generic (PLEG): container finished" podID="68decd37-fefc-41d4-8a7b-ab6b40e1096b" containerID="595ab4522be1caa69f9f627431bdeac961f950522b64127ef19a974115836192" exitCode=0 Sep 15 16:28:59.528850 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:59.528410 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" event={"ID":"68decd37-fefc-41d4-8a7b-ab6b40e1096b","Type":"ContainerDied","Data":"595ab4522be1caa69f9f627431bdeac961f950522b64127ef19a974115836192"} Sep 15 16:28:59.528850 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:59.528713 2558 scope.go:117] "RemoveContainer" containerID="595ab4522be1caa69f9f627431bdeac961f950522b64127ef19a974115836192" Sep 15 16:28:59.530049 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:59.530029 2558 generic.go:356] "Generic (PLEG): container finished" podID="4e672e0c-5a45-4d92-a24e-5bcba01e1adc" containerID="02b3bc5e7109e9a92ff7d1e230140b45685535edf3a8f5353b896df62e0e370c" exitCode=0 Sep 15 16:28:59.530158 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:59.530051 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" event={"ID":"4e672e0c-5a45-4d92-a24e-5bcba01e1adc","Type":"ContainerDied","Data":"02b3bc5e7109e9a92ff7d1e230140b45685535edf3a8f5353b896df62e0e370c"} Sep 15 16:28:59.530310 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:28:59.530298 2558 scope.go:117] "RemoveContainer" containerID="02b3bc5e7109e9a92ff7d1e230140b45685535edf3a8f5353b896df62e0e370c" Sep 15 16:29:00.536649 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:00.536616 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-72tnw" event={"ID":"68decd37-fefc-41d4-8a7b-ab6b40e1096b","Type":"ContainerStarted","Data":"05ac18b3a86f063e8597e1c0623f852d1e4a87ad69c4fe22ba6fc6ee94975742"} Sep 15 16:29:00.538543 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:00.538518 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-tjc6w" event={"ID":"4e672e0c-5a45-4d92-a24e-5bcba01e1adc","Type":"ContainerStarted","Data":"af52374dd440ccce17461a53c43b76729b64f346e261e761e3588f74d8d8f097"} Sep 15 16:29:15.561668 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.561628 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dc75f65f4-lw97r" podUID="57714f1a-b813-40ef-aa25-f0d37473d589" containerName="console" containerID="cri-o://4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69" gracePeriod=15 Sep 15 16:29:15.838770 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.838708 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:29:15.888364 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888338 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-serving-cert\") pod \"57714f1a-b813-40ef-aa25-f0d37473d589\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " Sep 15 16:29:15.888504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888380 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-console-config\") pod \"57714f1a-b813-40ef-aa25-f0d37473d589\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " Sep 15 16:29:15.888504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888397 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-oauth-serving-cert\") pod \"57714f1a-b813-40ef-aa25-f0d37473d589\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " Sep 15 16:29:15.888504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888436 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-oauth-config\") pod \"57714f1a-b813-40ef-aa25-f0d37473d589\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " Sep 15 16:29:15.888504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888463 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwngd\" (UniqueName: \"kubernetes.io/projected/57714f1a-b813-40ef-aa25-f0d37473d589-kube-api-access-cwngd\") pod \"57714f1a-b813-40ef-aa25-f0d37473d589\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " Sep 15 16:29:15.888504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888491 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-service-ca\") pod \"57714f1a-b813-40ef-aa25-f0d37473d589\" (UID: \"57714f1a-b813-40ef-aa25-f0d37473d589\") " Sep 15 16:29:15.888918 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888885 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "57714f1a-b813-40ef-aa25-f0d37473d589" (UID: "57714f1a-b813-40ef-aa25-f0d37473d589"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:29:15.889010 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888924 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-console-config" (OuterVolumeSpecName: "console-config") pod "57714f1a-b813-40ef-aa25-f0d37473d589" (UID: "57714f1a-b813-40ef-aa25-f0d37473d589"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:29:15.889010 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.888952 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-service-ca" (OuterVolumeSpecName: "service-ca") pod "57714f1a-b813-40ef-aa25-f0d37473d589" (UID: "57714f1a-b813-40ef-aa25-f0d37473d589"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:29:15.890684 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.890653 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "57714f1a-b813-40ef-aa25-f0d37473d589" (UID: "57714f1a-b813-40ef-aa25-f0d37473d589"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:15.890811 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.890679 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57714f1a-b813-40ef-aa25-f0d37473d589-kube-api-access-cwngd" (OuterVolumeSpecName: "kube-api-access-cwngd") pod "57714f1a-b813-40ef-aa25-f0d37473d589" (UID: "57714f1a-b813-40ef-aa25-f0d37473d589"). InnerVolumeSpecName "kube-api-access-cwngd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:29:15.890811 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.890763 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "57714f1a-b813-40ef-aa25-f0d37473d589" (UID: "57714f1a-b813-40ef-aa25-f0d37473d589"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:15.989903 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.989875 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-service-ca\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:15.989903 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.989898 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:15.989903 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.989908 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-console-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:15.990086 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.989916 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57714f1a-b813-40ef-aa25-f0d37473d589-oauth-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:15.990086 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.989934 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57714f1a-b813-40ef-aa25-f0d37473d589-console-oauth-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:15.990086 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:15.989943 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwngd\" (UniqueName: \"kubernetes.io/projected/57714f1a-b813-40ef-aa25-f0d37473d589-kube-api-access-cwngd\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:16.603845 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.603808 2558 generic.go:356] "Generic (PLEG): container finished" podID="57714f1a-b813-40ef-aa25-f0d37473d589" containerID="4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69" exitCode=0 Sep 15 16:29:16.603845 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.603850 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dc75f65f4-lw97r" Sep 15 16:29:16.604259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.603851 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc75f65f4-lw97r" event={"ID":"57714f1a-b813-40ef-aa25-f0d37473d589","Type":"ContainerDied","Data":"4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69"} Sep 15 16:29:16.604259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.603951 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dc75f65f4-lw97r" event={"ID":"57714f1a-b813-40ef-aa25-f0d37473d589","Type":"ContainerDied","Data":"80b78d37dadafe8a55a14eb6d424df4afc4aefa934655f5023a40c90aaf31e5c"} Sep 15 16:29:16.604259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.603969 2558 scope.go:117] "RemoveContainer" containerID="4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69" Sep 15 16:29:16.612519 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.612501 2558 scope.go:117] "RemoveContainer" containerID="4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69" Sep 15 16:29:16.612794 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:16.612775 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69\": container with ID starting with 4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69 not found: ID does not exist" containerID="4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69" Sep 15 16:29:16.612844 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.612803 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69"} err="failed to get container status \"4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69\": rpc error: code = NotFound desc = could not find container \"4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69\": container with ID starting with 4808483899a49365a00dd937b8d001be92a576b4c832eacdfa64887c1723cf69 not found: ID does not exist" Sep 15 16:29:16.624010 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.623987 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dc75f65f4-lw97r"] Sep 15 16:29:16.627822 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:16.627804 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dc75f65f4-lw97r"] Sep 15 16:29:17.726451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:17.726420 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57714f1a-b813-40ef-aa25-f0d37473d589" path="/var/lib/kubelet/pods/57714f1a-b813-40ef-aa25-f0d37473d589/volumes" Sep 15 16:29:27.055007 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.054965 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:29:27.055533 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.055356 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="alertmanager" containerID="cri-o://580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180" gracePeriod=120 Sep 15 16:29:27.055533 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.055384 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-metric" containerID="cri-o://2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023" gracePeriod=120 Sep 15 16:29:27.055533 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.055428 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="prom-label-proxy" containerID="cri-o://38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58" gracePeriod=120 Sep 15 16:29:27.055533 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.055457 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-web" containerID="cri-o://cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab" gracePeriod=120 Sep 15 16:29:27.055533 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.055463 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="config-reloader" containerID="cri-o://9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19" gracePeriod=120 Sep 15 16:29:27.055809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.055474 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy" containerID="cri-o://252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f" gracePeriod=120 Sep 15 16:29:27.648873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.648839 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58" exitCode=0 Sep 15 16:29:27.648873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.648866 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f" exitCode=0 Sep 15 16:29:27.648873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.648876 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19" exitCode=0 Sep 15 16:29:27.648873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.648883 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180" exitCode=0 Sep 15 16:29:27.649136 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.648907 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58"} Sep 15 16:29:27.649136 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.648978 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f"} Sep 15 16:29:27.649136 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.649005 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19"} Sep 15 16:29:27.649136 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:27.649031 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180"} Sep 15 16:29:28.316425 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.316399 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.400222 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400145 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400222 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400192 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8pk4\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-kube-api-access-k8pk4\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400233 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-config-out\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400265 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400299 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400451 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-metrics-client-ca\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400503 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-tls-assets\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400552 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-alertmanager-trusted-ca-bundle\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400582 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-alertmanager-main-db\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400609 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-config-volume\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400638 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400701 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-web-config\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400774 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-cluster-tls-config\") pod \"c3328151-caad-4540-be88-3493608431dc\" (UID: \"c3328151-caad-4540-be88-3493608431dc\") " Sep 15 16:29:28.400945 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.400925 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 16:29:28.401097 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.401072 2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-alertmanager-main-db\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.402374 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.401464 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:29:28.402374 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.402104 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:29:28.403300 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.403073 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.404275 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.404101 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.404408 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.404379 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-config-out" (OuterVolumeSpecName: "config-out") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 16:29:28.404476 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.404399 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.405338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.404566 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.405338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.404711 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-kube-api-access-k8pk4" (OuterVolumeSpecName: "kube-api-access-k8pk4") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "kube-api-access-k8pk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:29:28.405338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.404885 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:29:28.406014 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.405993 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-config-volume" (OuterVolumeSpecName: "config-volume") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.408269 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.408247 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.415409 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.415383 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-web-config" (OuterVolumeSpecName: "web-config") pod "c3328151-caad-4540-be88-3493608431dc" (UID: "c3328151-caad-4540-be88-3493608431dc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:29:28.501995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.501948 2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3328151-caad-4540-be88-3493608431dc-config-out\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.501995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.501991 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.501995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502002 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.501995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502012 2558 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-metrics-client-ca\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502020 2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-tls-assets\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502029 2558 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3328151-caad-4540-be88-3493608431dc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502038 2558 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-config-volume\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502046 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502057 2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-web-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502064 2558 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-cluster-tls-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502072 2558 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3328151-caad-4540-be88-3493608431dc-secret-alertmanager-main-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.502259 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.502081 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pk4\" (UniqueName: \"kubernetes.io/projected/c3328151-caad-4540-be88-3493608431dc-kube-api-access-k8pk4\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:29:28.655510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655429 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023" exitCode=0 Sep 15 16:29:28.655510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655452 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3328151-caad-4540-be88-3493608431dc" containerID="cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab" exitCode=0 Sep 15 16:29:28.655510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655494 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023"} Sep 15 16:29:28.655510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655507 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.655819 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655551 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab"} Sep 15 16:29:28.655819 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655579 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3328151-caad-4540-be88-3493608431dc","Type":"ContainerDied","Data":"cf6c8b3f9f86581a66bc8c53fdb52411f88bdefcd4edf8714cfdfb6c7aa9e4b9"} Sep 15 16:29:28.655819 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.655627 2558 scope.go:117] "RemoveContainer" containerID="38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58" Sep 15 16:29:28.663983 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.663967 2558 scope.go:117] "RemoveContainer" containerID="2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023" Sep 15 16:29:28.670913 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.670896 2558 scope.go:117] "RemoveContainer" containerID="252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f" Sep 15 16:29:28.679769 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.679749 2558 scope.go:117] "RemoveContainer" containerID="cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab" Sep 15 16:29:28.681588 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.681552 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:29:28.684118 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.684097 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:29:28.686982 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.686965 2558 scope.go:117] "RemoveContainer" containerID="9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19" Sep 15 16:29:28.693494 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.693476 2558 scope.go:117] "RemoveContainer" containerID="580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180" Sep 15 16:29:28.699983 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.699968 2558 scope.go:117] "RemoveContainer" containerID="5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a" Sep 15 16:29:28.706525 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.706504 2558 scope.go:117] "RemoveContainer" containerID="38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58" Sep 15 16:29:28.706895 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.706875 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58\": container with ID starting with 38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58 not found: ID does not exist" containerID="38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58" Sep 15 16:29:28.707005 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.706899 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58"} err="failed to get container status \"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58\": rpc error: code = NotFound desc = could not find container \"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58\": container with ID starting with 38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58 not found: ID does not exist" Sep 15 16:29:28.707005 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.706922 2558 scope.go:117] "RemoveContainer" containerID="2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023" Sep 15 16:29:28.707242 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.707217 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023\": container with ID starting with 2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023 not found: ID does not exist" containerID="2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023" Sep 15 16:29:28.707329 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.707246 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023"} err="failed to get container status \"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023\": rpc error: code = NotFound desc = could not find container \"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023\": container with ID starting with 2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023 not found: ID does not exist" Sep 15 16:29:28.707329 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.707266 2558 scope.go:117] "RemoveContainer" containerID="252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f" Sep 15 16:29:28.707574 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.707556 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f\": container with ID starting with 252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f not found: ID does not exist" containerID="252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f" Sep 15 16:29:28.707650 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.707582 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f"} err="failed to get container status \"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f\": rpc error: code = NotFound desc = could not find container \"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f\": container with ID starting with 252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f not found: ID does not exist" Sep 15 16:29:28.707650 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.707602 2558 scope.go:117] "RemoveContainer" containerID="cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab" Sep 15 16:29:28.707914 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.707899 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab\": container with ID starting with cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab not found: ID does not exist" containerID="cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab" Sep 15 16:29:28.707961 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.707918 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab"} err="failed to get container status \"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab\": rpc error: code = NotFound desc = could not find container \"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab\": container with ID starting with cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab not found: ID does not exist" Sep 15 16:29:28.707961 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.707932 2558 scope.go:117] "RemoveContainer" containerID="9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19" Sep 15 16:29:28.708192 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.708176 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19\": container with ID starting with 9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19 not found: ID does not exist" containerID="9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19" Sep 15 16:29:28.708239 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708194 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19"} err="failed to get container status \"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19\": rpc error: code = NotFound desc = could not find container \"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19\": container with ID starting with 9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19 not found: ID does not exist" Sep 15 16:29:28.708239 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708205 2558 scope.go:117] "RemoveContainer" containerID="580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180" Sep 15 16:29:28.708382 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708361 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:29:28.708457 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.708418 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180\": container with ID starting with 580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180 not found: ID does not exist" containerID="580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180" Sep 15 16:29:28.708457 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708435 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180"} err="failed to get container status \"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180\": rpc error: code = NotFound desc = could not find container \"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180\": container with ID starting with 580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180 not found: ID does not exist" Sep 15 16:29:28.708457 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708449 2558 scope.go:117] "RemoveContainer" containerID="5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a" Sep 15 16:29:28.708674 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708651 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-web" Sep 15 16:29:28.708674 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708670 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-web" Sep 15 16:29:28.708760 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708680 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="alertmanager" Sep 15 16:29:28.708760 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708686 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="alertmanager" Sep 15 16:29:28.708760 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708694 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="config-reloader" Sep 15 16:29:28.708760 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708701 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="config-reloader" Sep 15 16:29:28.708760 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708713 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57714f1a-b813-40ef-aa25-f0d37473d589" containerName="console" Sep 15 16:29:28.708760 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708719 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="57714f1a-b813-40ef-aa25-f0d37473d589" containerName="console" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708767 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="init-config-reloader" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708774 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="init-config-reloader" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708781 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="prom-label-proxy" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708788 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="prom-label-proxy" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708795 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" containerName="registry" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708800 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" containerName="registry" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708807 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708812 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708820 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-metric" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708826 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-metric" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:29:28.708718 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a\": container with ID starting with 5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a not found: ID does not exist" containerID="5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708864 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a"} err="failed to get container status \"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a\": rpc error: code = NotFound desc = could not find container \"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a\": container with ID starting with 5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a not found: ID does not exist" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708880 2558 scope.go:117] "RemoveContainer" containerID="38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708902 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="prom-label-proxy" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708912 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="alertmanager" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708918 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="config-reloader" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708925 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-web" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708931 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy-metric" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708938 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ffc94ac-2e9b-41cd-82be-a05528c51c25" containerName="registry" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708944 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="57714f1a-b813-40ef-aa25-f0d37473d589" containerName="console" Sep 15 16:29:28.709069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.708950 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3328151-caad-4540-be88-3493608431dc" containerName="kube-rbac-proxy" Sep 15 16:29:28.709828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709136 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58"} err="failed to get container status \"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58\": rpc error: code = NotFound desc = could not find container \"38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58\": container with ID starting with 38d5281f7c3353c192e2ac39d2e3859932681e909b28162040868c53f1f11c58 not found: ID does not exist" Sep 15 16:29:28.709828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709158 2558 scope.go:117] "RemoveContainer" containerID="2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023" Sep 15 16:29:28.709828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709411 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023"} err="failed to get container status \"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023\": rpc error: code = NotFound desc = could not find container \"2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023\": container with ID starting with 2630669ee6159ca79ea08ecf935c65d9c45c6f9582079574b50d6a9e69f92023 not found: ID does not exist" Sep 15 16:29:28.709828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709426 2558 scope.go:117] "RemoveContainer" containerID="252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f" Sep 15 16:29:28.709828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709646 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f"} err="failed to get container status \"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f\": rpc error: code = NotFound desc = could not find container \"252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f\": container with ID starting with 252a4787c9ff9959ea589d6fe022f73e9a329b2bd9166705c66cf7a0a7ab825f not found: ID does not exist" Sep 15 16:29:28.709828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709659 2558 scope.go:117] "RemoveContainer" containerID="cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab" Sep 15 16:29:28.710072 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709877 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab"} err="failed to get container status \"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab\": rpc error: code = NotFound desc = could not find container \"cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab\": container with ID starting with cce0ad8ab82f5cab40ed000af31419deec239d8e88f260a1cc11c7cc1ff93fab not found: ID does not exist" Sep 15 16:29:28.710072 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.709894 2558 scope.go:117] "RemoveContainer" containerID="9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19" Sep 15 16:29:28.710146 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.710130 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19"} err="failed to get container status \"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19\": rpc error: code = NotFound desc = could not find container \"9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19\": container with ID starting with 9ce49ee8e629b2349a61765fa477ba6952826f62f5090e316203f10b1fba0a19 not found: ID does not exist" Sep 15 16:29:28.710146 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.710142 2558 scope.go:117] "RemoveContainer" containerID="580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180" Sep 15 16:29:28.710356 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.710339 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180"} err="failed to get container status \"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180\": rpc error: code = NotFound desc = could not find container \"580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180\": container with ID starting with 580ff247a5a7ab31d7faf650a7020a61299548ce9c1e47a7e097af85a55ca180 not found: ID does not exist" Sep 15 16:29:28.710398 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.710357 2558 scope.go:117] "RemoveContainer" containerID="5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a" Sep 15 16:29:28.710565 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.710550 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a"} err="failed to get container status \"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a\": rpc error: code = NotFound desc = could not find container \"5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a\": container with ID starting with 5f9228bedea74298b654aa013ef51fe2af4e52e12a63f368880ec9763928573a not found: ID does not exist" Sep 15 16:29:28.712568 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.712553 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.715171 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715152 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 15 16:29:28.715256 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715153 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 15 16:29:28.715386 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715373 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t5gqm\"" Sep 15 16:29:28.715454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715392 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 15 16:29:28.715454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715392 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 15 16:29:28.715560 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715379 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 15 16:29:28.715698 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715683 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 15 16:29:28.715698 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715694 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 15 16:29:28.715837 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.715808 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 15 16:29:28.724878 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.723746 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 15 16:29:28.729701 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.729683 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:29:28.804449 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804404 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804449 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804448 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9fd7d9-5578-45d4-8f18-972235c6bec9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804471 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804492 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6vh\" (UniqueName: \"kubernetes.io/projected/be9fd7d9-5578-45d4-8f18-972235c6bec9-kube-api-access-gw6vh\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804508 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-config-volume\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804572 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9fd7d9-5578-45d4-8f18-972235c6bec9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804632 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-web-config\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804671 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804711 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9fd7d9-5578-45d4-8f18-972235c6bec9-config-out\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804801 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9fd7d9-5578-45d4-8f18-972235c6bec9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804834 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.804926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.804872 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9fd7d9-5578-45d4-8f18-972235c6bec9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.905781 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905683 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.905781 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905740 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9fd7d9-5578-45d4-8f18-972235c6bec9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.905781 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905764 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905784 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6vh\" (UniqueName: \"kubernetes.io/projected/be9fd7d9-5578-45d4-8f18-972235c6bec9-kube-api-access-gw6vh\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905802 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-config-volume\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905818 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9fd7d9-5578-45d4-8f18-972235c6bec9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905837 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-web-config\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.905998 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.906067 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9fd7d9-5578-45d4-8f18-972235c6bec9-config-out\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.906109 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.906134 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9fd7d9-5578-45d4-8f18-972235c6bec9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.906184 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906382 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.906224 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9fd7d9-5578-45d4-8f18-972235c6bec9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.906655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.906630 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be9fd7d9-5578-45d4-8f18-972235c6bec9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.908100 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908069 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be9fd7d9-5578-45d4-8f18-972235c6bec9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.908942 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908663 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be9fd7d9-5578-45d4-8f18-972235c6bec9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.908942 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908778 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.908942 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908874 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be9fd7d9-5578-45d4-8f18-972235c6bec9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.908942 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908885 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-config-volume\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.909174 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908967 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-web-config\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.909174 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.908990 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.909277 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.909192 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.909638 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.909617 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be9fd7d9-5578-45d4-8f18-972235c6bec9-config-out\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.910591 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.910570 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.910873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.910858 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be9fd7d9-5578-45d4-8f18-972235c6bec9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:28.914649 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:28.914633 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6vh\" (UniqueName: \"kubernetes.io/projected/be9fd7d9-5578-45d4-8f18-972235c6bec9-kube-api-access-gw6vh\") pod \"alertmanager-main-0\" (UID: \"be9fd7d9-5578-45d4-8f18-972235c6bec9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:29.028980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:29.028948 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 15 16:29:29.307502 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:29.307477 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 15 16:29:29.309005 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:29:29.308982 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9fd7d9_5578_45d4_8f18_972235c6bec9.slice/crio-d24359a53255a25b32ad44398a0b27250b183e493d1f014a33ac55d423c3a473 WatchSource:0}: Error finding container d24359a53255a25b32ad44398a0b27250b183e493d1f014a33ac55d423c3a473: Status 404 returned error can't find the container with id d24359a53255a25b32ad44398a0b27250b183e493d1f014a33ac55d423c3a473 Sep 15 16:29:29.661894 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:29.661813 2558 generic.go:356] "Generic (PLEG): container finished" podID="be9fd7d9-5578-45d4-8f18-972235c6bec9" containerID="b4d499e35d7ef9bd0f591e54191f03d147fbd02d9e82ccc0eaf938bb1b7f3362" exitCode=0 Sep 15 16:29:29.661894 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:29.661862 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerDied","Data":"b4d499e35d7ef9bd0f591e54191f03d147fbd02d9e82ccc0eaf938bb1b7f3362"} Sep 15 16:29:29.662287 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:29.661913 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"d24359a53255a25b32ad44398a0b27250b183e493d1f014a33ac55d423c3a473"} Sep 15 16:29:29.728181 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:29.728122 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3328151-caad-4540-be88-3493608431dc" path="/var/lib/kubelet/pods/c3328151-caad-4540-be88-3493608431dc/volumes" Sep 15 16:29:30.668338 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.668301 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"6514d525f90868740bdc89e02379d63b7e2cb21c5975594a4c1b184f31618fe5"} Sep 15 16:29:30.668836 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.668389 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"1927de03e915974902aee527e7a764513d7e48ceea276e346eb8ff4145ca909b"} Sep 15 16:29:30.668836 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.668486 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"56d16d77d4bfea0b7ccb2a7be08c12c492abc39c012bed2adbeeb9f65ed46ee1"} Sep 15 16:29:30.668836 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.668526 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"58640b9420842bdaeceb6c0aef945d826bdf176bf14eba20a3c9f8e284a9c21a"} Sep 15 16:29:30.668836 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.668565 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"6bf203d005f8b14a5ecf154bbca6a89a7fd0cefb724e9e16f6b103dbb219f48b"} Sep 15 16:29:30.668836 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.668611 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be9fd7d9-5578-45d4-8f18-972235c6bec9","Type":"ContainerStarted","Data":"518566af54f57aa526e56ee3d2627173bbd98ee4039a1177eef068cdd6cb4a0d"} Sep 15 16:29:30.696337 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:30.696288 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.680278797 podStartE2EDuration="2.69627157s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.992773ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:30.694423492 +0000 UTC m=+167.463705319" watchObservedRunningTime="2026-09-15 16:29:30.69627157 +0000 UTC m=+167.465553397" Sep 15 16:29:31.085313 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.085272 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-db6f84fdb-w9zxc"] Sep 15 16:29:31.087790 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.087766 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.090478 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.090456 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 15 16:29:31.090656 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.090634 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 15 16:29:31.090747 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.090710 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jndwn\"" Sep 15 16:29:31.090824 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.090710 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 15 16:29:31.091036 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.091013 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 15 16:29:31.091126 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.091033 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 15 16:29:31.095454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.095435 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 15 16:29:31.101880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.101857 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db6f84fdb-w9zxc"] Sep 15 16:29:31.231756 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231667 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-federate-client-tls\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231756 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231750 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-telemeter-client-tls\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231790 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231817 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-metrics-client-ca\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231872 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjzp\" (UniqueName: \"kubernetes.io/projected/2983aa89-38c9-4d25-8069-5d270cdf481f-kube-api-access-jsjzp\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231889 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-secret-telemeter-client\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231922 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.231996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.231941 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-serving-certs-ca-bundle\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332524 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332482 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332618 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332544 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-metrics-client-ca\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332618 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjzp\" (UniqueName: \"kubernetes.io/projected/2983aa89-38c9-4d25-8069-5d270cdf481f-kube-api-access-jsjzp\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332811 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332789 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-secret-telemeter-client\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332836 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332861 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-serving-certs-ca-bundle\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332983 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332895 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-federate-client-tls\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.332983 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.332913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-telemeter-client-tls\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.333472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.333440 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-metrics-client-ca\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.333590 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.333539 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-serving-certs-ca-bundle\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.333695 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.333667 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2983aa89-38c9-4d25-8069-5d270cdf481f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.335542 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.335489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.335810 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.335789 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-secret-telemeter-client\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.335857 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.335801 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-federate-client-tls\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.335889 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.335871 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2983aa89-38c9-4d25-8069-5d270cdf481f-telemeter-client-tls\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.344224 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.344199 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjzp\" (UniqueName: \"kubernetes.io/projected/2983aa89-38c9-4d25-8069-5d270cdf481f-kube-api-access-jsjzp\") pod \"telemeter-client-db6f84fdb-w9zxc\" (UID: \"2983aa89-38c9-4d25-8069-5d270cdf481f\") " pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.397557 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.397524 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" Sep 15 16:29:31.671229 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:31.671205 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-db6f84fdb-w9zxc"] Sep 15 16:29:31.672529 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:29:31.672501 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2983aa89_38c9_4d25_8069_5d270cdf481f.slice/crio-22f82aafa1d8522b9155dc17ba04f6afe662ae8f39b849f27bf6cafacf774226 WatchSource:0}: Error finding container 22f82aafa1d8522b9155dc17ba04f6afe662ae8f39b849f27bf6cafacf774226: Status 404 returned error can't find the container with id 22f82aafa1d8522b9155dc17ba04f6afe662ae8f39b849f27bf6cafacf774226 Sep 15 16:29:32.679488 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:32.679449 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" event={"ID":"2983aa89-38c9-4d25-8069-5d270cdf481f","Type":"ContainerStarted","Data":"22f82aafa1d8522b9155dc17ba04f6afe662ae8f39b849f27bf6cafacf774226"} Sep 15 16:29:33.685532 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.685501 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" event={"ID":"2983aa89-38c9-4d25-8069-5d270cdf481f","Type":"ContainerStarted","Data":"cb0d6f4333f100ac40f35c1e822652b88629798f893dd9efa723fe6f7c349e38"} Sep 15 16:29:33.685980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.685554 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" event={"ID":"2983aa89-38c9-4d25-8069-5d270cdf481f","Type":"ContainerStarted","Data":"87ff4d6538112daa9a03660943991093eb61731340359f2165515cafe7c107ff"} Sep 15 16:29:33.685980 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.685574 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" event={"ID":"2983aa89-38c9-4d25-8069-5d270cdf481f","Type":"ContainerStarted","Data":"e6f9155bb5ea943fc8d1ccd195b9ebfd4f918fb220340cd4f4ae63ea856d12a2"} Sep 15 16:29:33.712902 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.712851 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-db6f84fdb-w9zxc" podStartSLOduration=1.197763628 podStartE2EDuration="2.712836737s" totalImagesPullingTime="1.515073109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:33.711707772 +0000 UTC m=+170.480989600" watchObservedRunningTime="2026-09-15 16:29:33.712836737 +0000 UTC m=+170.482118563" Sep 15 16:29:33.807763 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.807713 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bd799fd-ml2qg"] Sep 15 16:29:33.810416 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.810392 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.825270 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.825245 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bd799fd-ml2qg"] Sep 15 16:29:33.953962 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.953878 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-serving-cert\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.953962 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.953916 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-oauth-serving-cert\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.954143 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.954016 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-service-ca\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.954143 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.954048 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwz4\" (UniqueName: \"kubernetes.io/projected/18ea4187-1690-4f05-bf53-5b3953efb24e-kube-api-access-xdwz4\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.954143 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.954123 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-trusted-ca-bundle\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.954245 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.954146 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-oauth-config\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:33.954245 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:33.954167 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-console-config\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055275 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055244 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-serving-cert\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055275 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055279 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-oauth-serving-cert\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055475 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055329 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-service-ca\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055475 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwz4\" (UniqueName: \"kubernetes.io/projected/18ea4187-1690-4f05-bf53-5b3953efb24e-kube-api-access-xdwz4\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055475 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055376 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-trusted-ca-bundle\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055475 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055401 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-oauth-config\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.055475 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.055422 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-console-config\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.056130 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.056096 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-oauth-serving-cert\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.056130 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.056122 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-console-config\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.056288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.056166 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-service-ca\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.056288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.056235 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-trusted-ca-bundle\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.058444 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.058420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-serving-cert\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.058526 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.058422 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-oauth-config\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.064130 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.064112 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwz4\" (UniqueName: \"kubernetes.io/projected/18ea4187-1690-4f05-bf53-5b3953efb24e-kube-api-access-xdwz4\") pod \"console-55bd799fd-ml2qg\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.121449 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.121413 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:34.387075 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.387044 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bd799fd-ml2qg"] Sep 15 16:29:34.390282 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:29:34.390247 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ea4187_1690_4f05_bf53_5b3953efb24e.slice/crio-325bcc1671fc2343f69652c5af73618b625685b03e71282553230f7ed6848ef6 WatchSource:0}: Error finding container 325bcc1671fc2343f69652c5af73618b625685b03e71282553230f7ed6848ef6: Status 404 returned error can't find the container with id 325bcc1671fc2343f69652c5af73618b625685b03e71282553230f7ed6848ef6 Sep 15 16:29:34.693056 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.692949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd799fd-ml2qg" event={"ID":"18ea4187-1690-4f05-bf53-5b3953efb24e","Type":"ContainerStarted","Data":"03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821"} Sep 15 16:29:34.693056 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.693005 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd799fd-ml2qg" event={"ID":"18ea4187-1690-4f05-bf53-5b3953efb24e","Type":"ContainerStarted","Data":"325bcc1671fc2343f69652c5af73618b625685b03e71282553230f7ed6848ef6"} Sep 15 16:29:34.711187 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:34.711140 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55bd799fd-ml2qg" podStartSLOduration=1.7111258230000002 podStartE2EDuration="1.711125823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:34.709954688 +0000 UTC m=+171.479236518" watchObservedRunningTime="2026-09-15 16:29:34.711125823 +0000 UTC m=+171.480407650" Sep 15 16:29:44.121539 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:44.121500 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:44.122010 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:44.121615 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:44.127702 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:44.127680 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:44.740043 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:44.740016 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:29:44.787297 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:29:44.787266 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b468b688c-l6ksj"] Sep 15 16:30:09.808142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:09.808095 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b468b688c-l6ksj" podUID="98a2aad4-981a-4cc4-9808-36a1bd1141e1" containerName="console" containerID="cri-o://ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd" gracePeriod=15 Sep 15 16:30:10.059022 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.058968 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:30:10.144827 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.144796 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-config\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.144981 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.144848 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-service-ca\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.144981 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.144873 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-oauth-serving-cert\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.144981 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.144913 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj47n\" (UniqueName: \"kubernetes.io/projected/98a2aad4-981a-4cc4-9808-36a1bd1141e1-kube-api-access-gj47n\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.144981 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.144940 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-oauth-config\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.145153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145005 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-trusted-ca-bundle\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.145153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145038 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-serving-cert\") pod \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\" (UID: \"98a2aad4-981a-4cc4-9808-36a1bd1141e1\") " Sep 15 16:30:10.145254 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145230 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-config" (OuterVolumeSpecName: "console-config") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:30:10.145317 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145253 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:30:10.145431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145397 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:30:10.145511 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145466 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.145511 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145486 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-service-ca\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.145613 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.145534 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:30:10.147156 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.147131 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a2aad4-981a-4cc4-9808-36a1bd1141e1-kube-api-access-gj47n" (OuterVolumeSpecName: "kube-api-access-gj47n") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "kube-api-access-gj47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:30:10.147240 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.147199 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:30:10.147295 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.147276 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98a2aad4-981a-4cc4-9808-36a1bd1141e1" (UID: "98a2aad4-981a-4cc4-9808-36a1bd1141e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:30:10.246139 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.246100 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-oauth-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.246139 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.246132 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj47n\" (UniqueName: \"kubernetes.io/projected/98a2aad4-981a-4cc4-9808-36a1bd1141e1-kube-api-access-gj47n\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.246139 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.246143 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-oauth-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.246367 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.246153 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98a2aad4-981a-4cc4-9808-36a1bd1141e1-trusted-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.246367 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.246161 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98a2aad4-981a-4cc4-9808-36a1bd1141e1-console-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:30:10.836830 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.836799 2558 generic.go:356] "Generic (PLEG): container finished" podID="98a2aad4-981a-4cc4-9808-36a1bd1141e1" containerID="ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd" exitCode=0 Sep 15 16:30:10.837288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.836842 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b468b688c-l6ksj" Sep 15 16:30:10.837288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.836889 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b468b688c-l6ksj" event={"ID":"98a2aad4-981a-4cc4-9808-36a1bd1141e1","Type":"ContainerDied","Data":"ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd"} Sep 15 16:30:10.837288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.836950 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b468b688c-l6ksj" event={"ID":"98a2aad4-981a-4cc4-9808-36a1bd1141e1","Type":"ContainerDied","Data":"d097bba3b788efed8671d496568d7be5efee39b1c7f351df9ba19af2d47bf861"} Sep 15 16:30:10.837288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.836983 2558 scope.go:117] "RemoveContainer" containerID="ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd" Sep 15 16:30:10.845937 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.845920 2558 scope.go:117] "RemoveContainer" containerID="ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd" Sep 15 16:30:10.846161 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:30:10.846141 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd\": container with ID starting with ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd not found: ID does not exist" containerID="ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd" Sep 15 16:30:10.846214 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.846182 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd"} err="failed to get container status \"ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd\": rpc error: code = NotFound desc = could not find container \"ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd\": container with ID starting with ee729701c077cd4fe9e145c8755b5d63e8c80f0b5fe0c8424e9b4af13f8b21cd not found: ID does not exist" Sep 15 16:30:10.857320 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.857293 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b468b688c-l6ksj"] Sep 15 16:30:10.860684 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:10.860663 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b468b688c-l6ksj"] Sep 15 16:30:11.727255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:11.727219 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a2aad4-981a-4cc4-9808-36a1bd1141e1" path="/var/lib/kubelet/pods/98a2aad4-981a-4cc4-9808-36a1bd1141e1/volumes" Sep 15 16:30:47.949882 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.949840 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69ff8d55c4-4gl22"] Sep 15 16:30:47.950305 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.950288 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a2aad4-981a-4cc4-9808-36a1bd1141e1" containerName="console" Sep 15 16:30:47.950350 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.950308 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a2aad4-981a-4cc4-9808-36a1bd1141e1" containerName="console" Sep 15 16:30:47.950418 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.950406 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a2aad4-981a-4cc4-9808-36a1bd1141e1" containerName="console" Sep 15 16:30:47.954670 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.954647 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958354 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958330 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-oauth-serving-cert\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958491 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958364 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-oauth-config\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958491 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958386 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-config\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958491 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958468 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-trusted-ca-bundle\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958514 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-service-ca\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958535 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8kj\" (UniqueName: \"kubernetes.io/projected/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-kube-api-access-db8kj\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.958655 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.958597 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-serving-cert\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:47.966260 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:47.966232 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69ff8d55c4-4gl22"] Sep 15 16:30:48.059152 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059114 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-service-ca\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059152 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8kj\" (UniqueName: \"kubernetes.io/projected/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-kube-api-access-db8kj\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059365 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-serving-cert\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059365 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-oauth-serving-cert\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059365 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-oauth-config\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059365 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059257 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-config\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059365 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059278 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-trusted-ca-bundle\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.059939 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.059906 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-service-ca\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.060079 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.060007 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-oauth-serving-cert\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.060079 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.060015 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-config\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.060241 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.060217 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-trusted-ca-bundle\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.061686 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.061657 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-oauth-config\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.061886 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.061868 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-serving-cert\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.068995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.068975 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8kj\" (UniqueName: \"kubernetes.io/projected/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-kube-api-access-db8kj\") pod \"console-69ff8d55c4-4gl22\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.263781 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.263743 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:48.534315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.534289 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69ff8d55c4-4gl22"] Sep 15 16:30:48.536981 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:30:48.536949 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a4a7c6_59e4_48ae_86b8_f4938bdf3052.slice/crio-28680b81dfe726f5757a45e560d434befaf3c52cfe7bb96513d7ac52ecef1633 WatchSource:0}: Error finding container 28680b81dfe726f5757a45e560d434befaf3c52cfe7bb96513d7ac52ecef1633: Status 404 returned error can't find the container with id 28680b81dfe726f5757a45e560d434befaf3c52cfe7bb96513d7ac52ecef1633 Sep 15 16:30:48.991458 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.991422 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff8d55c4-4gl22" event={"ID":"67a4a7c6-59e4-48ae-86b8-f4938bdf3052","Type":"ContainerStarted","Data":"22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3"} Sep 15 16:30:48.991893 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:48.991473 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff8d55c4-4gl22" event={"ID":"67a4a7c6-59e4-48ae-86b8-f4938bdf3052","Type":"ContainerStarted","Data":"28680b81dfe726f5757a45e560d434befaf3c52cfe7bb96513d7ac52ecef1633"} Sep 15 16:30:49.012975 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:49.012933 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69ff8d55c4-4gl22" podStartSLOduration=2.012917643 podStartE2EDuration="2.012917643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:30:49.011966372 +0000 UTC m=+245.781248234" watchObservedRunningTime="2026-09-15 16:30:49.012917643 +0000 UTC m=+245.782199472" Sep 15 16:30:58.264472 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:58.264428 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:58.264957 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:58.264521 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:58.269334 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:58.269311 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:59.035320 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:59.035289 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:30:59.085714 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:30:59.085682 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bd799fd-ml2qg"] Sep 15 16:31:24.110067 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.109998 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55bd799fd-ml2qg" podUID="18ea4187-1690-4f05-bf53-5b3953efb24e" containerName="console" containerID="cri-o://03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821" gracePeriod=15 Sep 15 16:31:24.344561 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.344534 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:31:24.468996 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.468977 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-oauth-config\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469018 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-trusted-ca-bundle\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469051 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-serving-cert\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469068 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-oauth-serving-cert\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469101 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdwz4\" (UniqueName: \"kubernetes.io/projected/18ea4187-1690-4f05-bf53-5b3953efb24e-kube-api-access-xdwz4\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469115 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-console-config\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469377 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469158 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-service-ca\") pod \"18ea4187-1690-4f05-bf53-5b3953efb24e\" (UID: \"18ea4187-1690-4f05-bf53-5b3953efb24e\") " Sep 15 16:31:24.469542 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469508 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:31:24.469661 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469595 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:31:24.469661 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469616 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-console-config" (OuterVolumeSpecName: "console-config") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:31:24.469661 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.469635 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-service-ca" (OuterVolumeSpecName: "service-ca") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:31:24.471276 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.471253 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ea4187-1690-4f05-bf53-5b3953efb24e-kube-api-access-xdwz4" (OuterVolumeSpecName: "kube-api-access-xdwz4") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "kube-api-access-xdwz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:31:24.471669 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.471646 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:31:24.471716 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.471656 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "18ea4187-1690-4f05-bf53-5b3953efb24e" (UID: "18ea4187-1690-4f05-bf53-5b3953efb24e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:31:24.570121 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570085 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-oauth-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:24.570121 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570113 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-trusted-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:24.570121 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570122 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18ea4187-1690-4f05-bf53-5b3953efb24e-console-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:24.570121 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570130 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-oauth-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:24.570382 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570139 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwz4\" (UniqueName: \"kubernetes.io/projected/18ea4187-1690-4f05-bf53-5b3953efb24e-kube-api-access-xdwz4\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:24.570382 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570148 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-console-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:24.570382 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:24.570162 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18ea4187-1690-4f05-bf53-5b3953efb24e-service-ca\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:31:25.138141 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.138105 2558 generic.go:356] "Generic (PLEG): container finished" podID="18ea4187-1690-4f05-bf53-5b3953efb24e" containerID="03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821" exitCode=0 Sep 15 16:31:25.138563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.138146 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bd799fd-ml2qg" Sep 15 16:31:25.138563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.138162 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd799fd-ml2qg" event={"ID":"18ea4187-1690-4f05-bf53-5b3953efb24e","Type":"ContainerDied","Data":"03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821"} Sep 15 16:31:25.138563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.138210 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bd799fd-ml2qg" event={"ID":"18ea4187-1690-4f05-bf53-5b3953efb24e","Type":"ContainerDied","Data":"325bcc1671fc2343f69652c5af73618b625685b03e71282553230f7ed6848ef6"} Sep 15 16:31:25.138563 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.138233 2558 scope.go:117] "RemoveContainer" containerID="03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821" Sep 15 16:31:25.147315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.147298 2558 scope.go:117] "RemoveContainer" containerID="03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821" Sep 15 16:31:25.147600 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:31:25.147583 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821\": container with ID starting with 03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821 not found: ID does not exist" containerID="03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821" Sep 15 16:31:25.147648 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.147608 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821"} err="failed to get container status \"03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821\": rpc error: code = NotFound desc = could not find container \"03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821\": container with ID starting with 03a6f64d4be2135d3ee0675b79e44f9f4c2fc6594f8c06745f58d1f3a0222821 not found: ID does not exist" Sep 15 16:31:25.160225 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.160204 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bd799fd-ml2qg"] Sep 15 16:31:25.164094 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.164074 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55bd799fd-ml2qg"] Sep 15 16:31:25.727103 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:25.727068 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ea4187-1690-4f05-bf53-5b3953efb24e" path="/var/lib/kubelet/pods/18ea4187-1690-4f05-bf53-5b3953efb24e/volumes" Sep 15 16:31:43.609634 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:43.609600 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:31:43.612080 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:43.612051 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:31:43.623263 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:31:43.623245 2558 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 16:33:37.563476 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.563443 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d6ff9cf4-q6txr"] Sep 15 16:33:37.563970 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.563774 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ea4187-1690-4f05-bf53-5b3953efb24e" containerName="console" Sep 15 16:33:37.563970 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.563787 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ea4187-1690-4f05-bf53-5b3953efb24e" containerName="console" Sep 15 16:33:37.563970 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.563832 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ea4187-1690-4f05-bf53-5b3953efb24e" containerName="console" Sep 15 16:33:37.566560 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.566535 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.587562 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.587534 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d6ff9cf4-q6txr"] Sep 15 16:33:37.648564 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648523 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-trusted-ca-bundle\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.648764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648574 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-oauth-serving-cert\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.648764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648608 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20539f6b-115b-497f-afea-fa6913c6c1be-console-oauth-config\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.648764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648666 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj4b5\" (UniqueName: \"kubernetes.io/projected/20539f6b-115b-497f-afea-fa6913c6c1be-kube-api-access-dj4b5\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.648764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648697 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-service-ca\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.648764 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648758 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20539f6b-115b-497f-afea-fa6913c6c1be-console-serving-cert\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.648977 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.648809 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-console-config\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749149 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-trusted-ca-bundle\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-oauth-serving-cert\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749207 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20539f6b-115b-497f-afea-fa6913c6c1be-console-oauth-config\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749244 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj4b5\" (UniqueName: \"kubernetes.io/projected/20539f6b-115b-497f-afea-fa6913c6c1be-kube-api-access-dj4b5\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-service-ca\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749285 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20539f6b-115b-497f-afea-fa6913c6c1be-console-serving-cert\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.749446 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.749308 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-console-config\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.750080 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.750049 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-oauth-serving-cert\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.750227 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.750098 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-console-config\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.750227 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.750114 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-service-ca\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.750227 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.750163 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20539f6b-115b-497f-afea-fa6913c6c1be-trusted-ca-bundle\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.751922 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.751898 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20539f6b-115b-497f-afea-fa6913c6c1be-console-oauth-config\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.752014 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.751942 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20539f6b-115b-497f-afea-fa6913c6c1be-console-serving-cert\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.757045 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.757019 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj4b5\" (UniqueName: \"kubernetes.io/projected/20539f6b-115b-497f-afea-fa6913c6c1be-kube-api-access-dj4b5\") pod \"console-9d6ff9cf4-q6txr\" (UID: \"20539f6b-115b-497f-afea-fa6913c6c1be\") " pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:37.876593 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:37.876512 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:38.151056 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:38.151031 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d6ff9cf4-q6txr"] Sep 15 16:33:38.153313 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:33:38.153285 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20539f6b_115b_497f_afea_fa6913c6c1be.slice/crio-f5002ef8718acf9a441425285766bc5ae3b314743e58cc01b9a16a517fca7756 WatchSource:0}: Error finding container f5002ef8718acf9a441425285766bc5ae3b314743e58cc01b9a16a517fca7756: Status 404 returned error can't find the container with id f5002ef8718acf9a441425285766bc5ae3b314743e58cc01b9a16a517fca7756 Sep 15 16:33:38.657526 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:38.657490 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d6ff9cf4-q6txr" event={"ID":"20539f6b-115b-497f-afea-fa6913c6c1be","Type":"ContainerStarted","Data":"652bff55b36e6aef2d1313259842b3dc1c568fe057f6abec04eb2560025e2e68"} Sep 15 16:33:38.657526 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:38.657541 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d6ff9cf4-q6txr" event={"ID":"20539f6b-115b-497f-afea-fa6913c6c1be","Type":"ContainerStarted","Data":"f5002ef8718acf9a441425285766bc5ae3b314743e58cc01b9a16a517fca7756"} Sep 15 16:33:38.679843 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:38.679794 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d6ff9cf4-q6txr" podStartSLOduration=1.679775467 podStartE2EDuration="1.679775467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:33:38.677473078 +0000 UTC m=+415.446754906" watchObservedRunningTime="2026-09-15 16:33:38.679775467 +0000 UTC m=+415.449057364" Sep 15 16:33:45.261708 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.261673 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g"] Sep 15 16:33:45.265455 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.265432 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.268872 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.268856 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-shjpz\"" Sep 15 16:33:45.269150 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.269135 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 15 16:33:45.269985 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.269969 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 15 16:33:45.276715 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.276687 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g"] Sep 15 16:33:45.304820 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.304795 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.304922 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.304857 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.304973 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.304914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrpw\" (UniqueName: \"kubernetes.io/projected/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-kube-api-access-6zrpw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.405455 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.405417 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.405455 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.405463 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrpw\" (UniqueName: \"kubernetes.io/projected/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-kube-api-access-6zrpw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.405689 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.405527 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.405863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.405843 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.405916 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.405871 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.416195 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.416166 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrpw\" (UniqueName: \"kubernetes.io/projected/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-kube-api-access-6zrpw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.574592 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.574505 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:33:45.844133 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.844115 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g"] Sep 15 16:33:45.846245 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:33:45.846218 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3297aa_36fa_48e3_952f_2d8cb16d6e7b.slice/crio-e3cde9529014246a8d1fb13c7320e74a6cd76c948982b945d94eb619bca1d02f WatchSource:0}: Error finding container e3cde9529014246a8d1fb13c7320e74a6cd76c948982b945d94eb619bca1d02f: Status 404 returned error can't find the container with id e3cde9529014246a8d1fb13c7320e74a6cd76c948982b945d94eb619bca1d02f Sep 15 16:33:45.848375 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:45.848356 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 16:33:46.691054 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:46.691016 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" event={"ID":"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b","Type":"ContainerStarted","Data":"e3cde9529014246a8d1fb13c7320e74a6cd76c948982b945d94eb619bca1d02f"} Sep 15 16:33:47.877108 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:47.877014 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:47.877510 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:47.877114 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:47.882578 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:47.882548 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:48.704001 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:48.703975 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d6ff9cf4-q6txr" Sep 15 16:33:48.764549 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:48.764514 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69ff8d55c4-4gl22"] Sep 15 16:33:52.719439 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:52.719400 2558 generic.go:356] "Generic (PLEG): container finished" podID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerID="5368d1b99625fcb11069162aa78df2cb086482de2d94225f600ff4ff279f418c" exitCode=0 Sep 15 16:33:52.719822 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:52.719483 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" event={"ID":"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b","Type":"ContainerDied","Data":"5368d1b99625fcb11069162aa78df2cb086482de2d94225f600ff4ff279f418c"} Sep 15 16:33:55.738861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:55.738823 2558 generic.go:356] "Generic (PLEG): container finished" podID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerID="ba090ca10723879a184113b0ce88c621189223fce07e534d0911efb8e3967d5e" exitCode=0 Sep 15 16:33:55.739330 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:33:55.738888 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" event={"ID":"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b","Type":"ContainerDied","Data":"ba090ca10723879a184113b0ce88c621189223fce07e534d0911efb8e3967d5e"} Sep 15 16:34:03.774863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:03.774831 2558 generic.go:356] "Generic (PLEG): container finished" podID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerID="53270607bbbbb8d143d5df250e91a7c579ad67f33c37467674a9678d2d3c0425" exitCode=0 Sep 15 16:34:03.775242 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:03.774898 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" event={"ID":"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b","Type":"ContainerDied","Data":"53270607bbbbb8d143d5df250e91a7c579ad67f33c37467674a9678d2d3c0425"} Sep 15 16:34:04.908931 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.908904 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:34:04.978878 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.978846 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-util\") pod \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " Sep 15 16:34:04.979027 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.978896 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zrpw\" (UniqueName: \"kubernetes.io/projected/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-kube-api-access-6zrpw\") pod \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " Sep 15 16:34:04.979027 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.978934 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-bundle\") pod \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\" (UID: \"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b\") " Sep 15 16:34:04.979571 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.979544 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-bundle" (OuterVolumeSpecName: "bundle") pod "2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" (UID: "2e3297aa-36fa-48e3-952f-2d8cb16d6e7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 16:34:04.981092 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.981053 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-kube-api-access-6zrpw" (OuterVolumeSpecName: "kube-api-access-6zrpw") pod "2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" (UID: "2e3297aa-36fa-48e3-952f-2d8cb16d6e7b"). InnerVolumeSpecName "kube-api-access-6zrpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:34:04.983783 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:04.983760 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-util" (OuterVolumeSpecName: "util") pod "2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" (UID: "2e3297aa-36fa-48e3-952f-2d8cb16d6e7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 16:34:05.080077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:05.080008 2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-util\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:05.080077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:05.080031 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrpw\" (UniqueName: \"kubernetes.io/projected/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-kube-api-access-6zrpw\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:05.080077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:05.080041 2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2e3297aa-36fa-48e3-952f-2d8cb16d6e7b-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:05.785153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:05.785124 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" Sep 15 16:34:05.785276 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:05.785152 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a94q8g" event={"ID":"2e3297aa-36fa-48e3-952f-2d8cb16d6e7b","Type":"ContainerDied","Data":"e3cde9529014246a8d1fb13c7320e74a6cd76c948982b945d94eb619bca1d02f"} Sep 15 16:34:05.785276 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:05.785194 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3cde9529014246a8d1fb13c7320e74a6cd76c948982b945d94eb619bca1d02f" Sep 15 16:34:13.284499 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284468 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx"] Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284767 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="util" Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284777 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="util" Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284793 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="pull" Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284799 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="pull" Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284808 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="extract" Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284813 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="extract" Sep 15 16:34:13.284986 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.284866 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e3297aa-36fa-48e3-952f-2d8cb16d6e7b" containerName="extract" Sep 15 16:34:13.286634 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.286619 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.291451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.291426 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 15 16:34:13.291604 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.291453 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 15 16:34:13.291604 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.291454 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 15 16:34:13.291604 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.291454 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-jzzsb\"" Sep 15 16:34:13.314598 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.314570 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx"] Sep 15 16:34:13.450832 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.450797 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6bb4c24d-6cb0-416e-b0a2-72721d890027-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lw2mx\" (UID: \"6bb4c24d-6cb0-416e-b0a2-72721d890027\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.451015 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.450841 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-598wz\" (UniqueName: \"kubernetes.io/projected/6bb4c24d-6cb0-416e-b0a2-72721d890027-kube-api-access-598wz\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lw2mx\" (UID: \"6bb4c24d-6cb0-416e-b0a2-72721d890027\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.551218 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.551144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-598wz\" (UniqueName: \"kubernetes.io/projected/6bb4c24d-6cb0-416e-b0a2-72721d890027-kube-api-access-598wz\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lw2mx\" (UID: \"6bb4c24d-6cb0-416e-b0a2-72721d890027\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.551218 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.551214 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6bb4c24d-6cb0-416e-b0a2-72721d890027-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lw2mx\" (UID: \"6bb4c24d-6cb0-416e-b0a2-72721d890027\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.553450 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.553425 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6bb4c24d-6cb0-416e-b0a2-72721d890027-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lw2mx\" (UID: \"6bb4c24d-6cb0-416e-b0a2-72721d890027\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.588678 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.588650 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-598wz\" (UniqueName: \"kubernetes.io/projected/6bb4c24d-6cb0-416e-b0a2-72721d890027-kube-api-access-598wz\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lw2mx\" (UID: \"6bb4c24d-6cb0-416e-b0a2-72721d890027\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.596470 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.596444 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:13.791429 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.791390 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-69ff8d55c4-4gl22" podUID="67a4a7c6-59e4-48ae-86b8-f4938bdf3052" containerName="console" containerID="cri-o://22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3" gracePeriod=15 Sep 15 16:34:13.885468 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:13.885440 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx"] Sep 15 16:34:13.887646 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:34:13.887618 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb4c24d_6cb0_416e_b0a2_72721d890027.slice/crio-c09397055fa4476e767181dee5ba996d7b4cc955c4e53e99e0fab68dcc7fbdd7 WatchSource:0}: Error finding container c09397055fa4476e767181dee5ba996d7b4cc955c4e53e99e0fab68dcc7fbdd7: Status 404 returned error can't find the container with id c09397055fa4476e767181dee5ba996d7b4cc955c4e53e99e0fab68dcc7fbdd7 Sep 15 16:34:14.017658 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.017633 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:34:14.156766 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156676 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-oauth-serving-cert\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.156766 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156713 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-trusted-ca-bundle\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.156766 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156757 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-serving-cert\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.157026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156791 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db8kj\" (UniqueName: \"kubernetes.io/projected/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-kube-api-access-db8kj\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.157026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156836 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-config\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.157026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156860 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-service-ca\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.157026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.156934 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-oauth-config\") pod \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\" (UID: \"67a4a7c6-59e4-48ae-86b8-f4938bdf3052\") " Sep 15 16:34:14.157228 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157211 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:34:14.157292 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157237 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-config" (OuterVolumeSpecName: "console-config") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:34:14.157292 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157251 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:34:14.157431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157379 2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-oauth-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.157431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157399 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-trusted-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.157431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157415 2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.157544 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.157450 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-service-ca" (OuterVolumeSpecName: "service-ca") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:34:14.158944 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.158926 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:34:14.159403 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.159375 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:34:14.159444 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.159387 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-kube-api-access-db8kj" (OuterVolumeSpecName: "kube-api-access-db8kj") pod "67a4a7c6-59e4-48ae-86b8-f4938bdf3052" (UID: "67a4a7c6-59e4-48ae-86b8-f4938bdf3052"). InnerVolumeSpecName "kube-api-access-db8kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:34:14.258623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.258588 2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-serving-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.258623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.258616 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8kj\" (UniqueName: \"kubernetes.io/projected/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-kube-api-access-db8kj\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.258623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.258626 2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-service-ca\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.258623 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.258634 2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67a4a7c6-59e4-48ae-86b8-f4938bdf3052-console-oauth-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:34:14.830254 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.830162 2558 generic.go:356] "Generic (PLEG): container finished" podID="67a4a7c6-59e4-48ae-86b8-f4938bdf3052" containerID="22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3" exitCode=0 Sep 15 16:34:14.830254 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.830223 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69ff8d55c4-4gl22" Sep 15 16:34:14.830800 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.830250 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff8d55c4-4gl22" event={"ID":"67a4a7c6-59e4-48ae-86b8-f4938bdf3052","Type":"ContainerDied","Data":"22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3"} Sep 15 16:34:14.830800 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.830303 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69ff8d55c4-4gl22" event={"ID":"67a4a7c6-59e4-48ae-86b8-f4938bdf3052","Type":"ContainerDied","Data":"28680b81dfe726f5757a45e560d434befaf3c52cfe7bb96513d7ac52ecef1633"} Sep 15 16:34:14.830800 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.830330 2558 scope.go:117] "RemoveContainer" containerID="22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3" Sep 15 16:34:14.832272 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.832240 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" event={"ID":"6bb4c24d-6cb0-416e-b0a2-72721d890027","Type":"ContainerStarted","Data":"c09397055fa4476e767181dee5ba996d7b4cc955c4e53e99e0fab68dcc7fbdd7"} Sep 15 16:34:14.840401 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.840376 2558 scope.go:117] "RemoveContainer" containerID="22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3" Sep 15 16:34:14.840861 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:34:14.840691 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3\": container with ID starting with 22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3 not found: ID does not exist" containerID="22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3" Sep 15 16:34:14.840861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.840719 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3"} err="failed to get container status \"22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3\": rpc error: code = NotFound desc = could not find container \"22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3\": container with ID starting with 22a5981a6f6dccd4da7c60332345523edbe1d1a1d4b16518a6c06cee4ca9ccb3 not found: ID does not exist" Sep 15 16:34:14.856429 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.856395 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69ff8d55c4-4gl22"] Sep 15 16:34:14.862615 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:14.862595 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69ff8d55c4-4gl22"] Sep 15 16:34:15.728398 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:15.728367 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a4a7c6-59e4-48ae-86b8-f4938bdf3052" path="/var/lib/kubelet/pods/67a4a7c6-59e4-48ae-86b8-f4938bdf3052/volumes" Sep 15 16:34:16.845354 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:16.845273 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" event={"ID":"6bb4c24d-6cb0-416e-b0a2-72721d890027","Type":"ContainerStarted","Data":"42f611d5806fe2c5963e4c712956752507537577d26d997c94dc741e9051e403"} Sep 15 16:34:16.845817 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:16.845465 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:16.874591 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:16.874547 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" podStartSLOduration=1.197977163 podStartE2EDuration="3.87453244s" totalImagesPullingTime="2.676555277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:16.873254414 +0000 UTC m=+453.642536241" watchObservedRunningTime="2026-09-15 16:34:16.87453244 +0000 UTC m=+453.643814267" Sep 15 16:34:17.505806 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.505780 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb"] Sep 15 16:34:17.506096 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.506084 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a4a7c6-59e4-48ae-86b8-f4938bdf3052" containerName="console" Sep 15 16:34:17.506096 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.506097 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a4a7c6-59e4-48ae-86b8-f4938bdf3052" containerName="console" Sep 15 16:34:17.506185 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.506163 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a4a7c6-59e4-48ae-86b8-f4938bdf3052" containerName="console" Sep 15 16:34:17.508210 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.508195 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.511155 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.511126 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 15 16:34:17.511301 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.511200 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 15 16:34:17.511379 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.511311 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-hh427\"" Sep 15 16:34:17.517625 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.517602 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb"] Sep 15 16:34:17.588241 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.588208 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwzq\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-kube-api-access-nfwzq\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.588378 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.588246 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.588378 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.588336 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e4c0ef50-ad16-4224-bce9-89091373bce9-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.688813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.688774 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e4c0ef50-ad16-4224-bce9-89091373bce9-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.688942 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.688879 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwzq\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-kube-api-access-nfwzq\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.688942 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.688909 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.689030 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:34:17.689010 2558 secret.go:281] references non-existent secret key: tls.crt Sep 15 16:34:17.689030 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:34:17.689022 2558 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Sep 15 16:34:17.689093 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:34:17.689037 2558 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb: references non-existent secret key: tls.crt Sep 15 16:34:17.689126 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:34:17.689091 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-certificates podName:e4c0ef50-ad16-4224-bce9-89091373bce9 nodeName:}" failed. No retries permitted until 2026-09-15 16:34:18.189074559 +0000 UTC m=+454.958356364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-certificates") pod "keda-metrics-apiserver-77857b7b98-lzcwb" (UID: "e4c0ef50-ad16-4224-bce9-89091373bce9") : references non-existent secret key: tls.crt Sep 15 16:34:17.689126 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.689092 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e4c0ef50-ad16-4224-bce9-89091373bce9-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:17.699646 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:17.699629 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwzq\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-kube-api-access-nfwzq\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:18.007099 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.006434 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9lb8k"] Sep 15 16:34:18.010204 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.010183 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.012486 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.012466 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 15 16:34:18.016001 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.015981 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9lb8k"] Sep 15 16:34:18.092562 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.092533 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9w8z\" (UniqueName: \"kubernetes.io/projected/00acca91-72d0-4b60-ad19-d0afabfd8349-kube-api-access-r9w8z\") pod \"keda-admission-6c4d864b86-9lb8k\" (UID: \"00acca91-72d0-4b60-ad19-d0afabfd8349\") " pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.092689 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.092571 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00acca91-72d0-4b60-ad19-d0afabfd8349-certificates\") pod \"keda-admission-6c4d864b86-9lb8k\" (UID: \"00acca91-72d0-4b60-ad19-d0afabfd8349\") " pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.193876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.193845 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9w8z\" (UniqueName: \"kubernetes.io/projected/00acca91-72d0-4b60-ad19-d0afabfd8349-kube-api-access-r9w8z\") pod \"keda-admission-6c4d864b86-9lb8k\" (UID: \"00acca91-72d0-4b60-ad19-d0afabfd8349\") " pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.194022 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.193891 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00acca91-72d0-4b60-ad19-d0afabfd8349-certificates\") pod \"keda-admission-6c4d864b86-9lb8k\" (UID: \"00acca91-72d0-4b60-ad19-d0afabfd8349\") " pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.194022 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.193939 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:18.196430 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.196406 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e4c0ef50-ad16-4224-bce9-89091373bce9-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lzcwb\" (UID: \"e4c0ef50-ad16-4224-bce9-89091373bce9\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:18.196522 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.196440 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00acca91-72d0-4b60-ad19-d0afabfd8349-certificates\") pod \"keda-admission-6c4d864b86-9lb8k\" (UID: \"00acca91-72d0-4b60-ad19-d0afabfd8349\") " pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.204512 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.204484 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9w8z\" (UniqueName: \"kubernetes.io/projected/00acca91-72d0-4b60-ad19-d0afabfd8349-kube-api-access-r9w8z\") pod \"keda-admission-6c4d864b86-9lb8k\" (UID: \"00acca91-72d0-4b60-ad19-d0afabfd8349\") " pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.321673 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.321585 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:18.419131 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.419099 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:18.594873 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.594828 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9lb8k"] Sep 15 16:34:18.597116 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:34:18.597090 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00acca91_72d0_4b60_ad19_d0afabfd8349.slice/crio-8d01c94eb131d6355971347b40ea0ec30069442187e2a9de698457dd7ed6ae00 WatchSource:0}: Error finding container 8d01c94eb131d6355971347b40ea0ec30069442187e2a9de698457dd7ed6ae00: Status 404 returned error can't find the container with id 8d01c94eb131d6355971347b40ea0ec30069442187e2a9de698457dd7ed6ae00 Sep 15 16:34:18.687376 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.687353 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb"] Sep 15 16:34:18.689233 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:34:18.689206 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c0ef50_ad16_4224_bce9_89091373bce9.slice/crio-c3e041fb3bcf4122a12042b16eec76bb7879290f55ca5bc8bfd3100dca318fd4 WatchSource:0}: Error finding container c3e041fb3bcf4122a12042b16eec76bb7879290f55ca5bc8bfd3100dca318fd4: Status 404 returned error can't find the container with id c3e041fb3bcf4122a12042b16eec76bb7879290f55ca5bc8bfd3100dca318fd4 Sep 15 16:34:18.856027 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.855950 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" event={"ID":"e4c0ef50-ad16-4224-bce9-89091373bce9","Type":"ContainerStarted","Data":"c3e041fb3bcf4122a12042b16eec76bb7879290f55ca5bc8bfd3100dca318fd4"} Sep 15 16:34:18.857130 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:18.857106 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" event={"ID":"00acca91-72d0-4b60-ad19-d0afabfd8349","Type":"ContainerStarted","Data":"8d01c94eb131d6355971347b40ea0ec30069442187e2a9de698457dd7ed6ae00"} Sep 15 16:34:20.870362 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:20.870326 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" event={"ID":"00acca91-72d0-4b60-ad19-d0afabfd8349","Type":"ContainerStarted","Data":"906a369a554d032e5ed985d13753f85e401f6e54a2bc64a80089a4c8fc71d4ae"} Sep 15 16:34:20.870861 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:20.870492 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:34:20.888110 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:20.888064 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" podStartSLOduration=2.302889621 podStartE2EDuration="3.888045201s" totalImagesPullingTime="1.58515558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:20.88680559 +0000 UTC m=+457.656087417" watchObservedRunningTime="2026-09-15 16:34:20.888045201 +0000 UTC m=+457.657327031" Sep 15 16:34:21.877083 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:21.877046 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" event={"ID":"e4c0ef50-ad16-4224-bce9-89091373bce9","Type":"ContainerStarted","Data":"60f41f6d866d9f38a3c2fd8e2d7c3d319a44204df05614bca53a12db1b4f03d8"} Sep 15 16:34:21.877525 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:21.877319 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:21.891240 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:21.891182 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" podStartSLOduration=2.421163184 podStartE2EDuration="4.89116152s" totalImagesPullingTime="2.469998336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:21.890893215 +0000 UTC m=+458.660175054" watchObservedRunningTime="2026-09-15 16:34:21.89116152 +0000 UTC m=+458.660443348" Sep 15 16:34:32.886172 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:32.886144 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lzcwb" Sep 15 16:34:37.852897 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:37.852863 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lw2mx" Sep 15 16:34:41.879784 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:34:41.879755 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-9lb8k" Sep 15 16:35:34.972463 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.972433 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-plx4c"] Sep 15 16:35:34.974687 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.974669 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:34.977509 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.977488 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 15 16:35:34.977509 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.977504 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2rw48\"" Sep 15 16:35:34.978372 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.978354 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 15 16:35:34.978424 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.978378 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 15 16:35:34.987609 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:34.987588 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-plx4c"] Sep 15 16:35:35.016645 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.016624 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-6lk77"] Sep 15 16:35:35.018700 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.018685 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.021719 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.021701 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dtbhk\"" Sep 15 16:35:35.022012 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.021995 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 15 16:35:35.034878 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.034852 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6lk77"] Sep 15 16:35:35.046876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.046852 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4537a962-0de7-4494-93af-46ce9698662b-data\") pod \"seaweedfs-64568bcd49-6lk77\" (UID: \"4537a962-0de7-4494-93af-46ce9698662b\") " pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.046965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.046893 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv6rf\" (UniqueName: \"kubernetes.io/projected/2fdba674-1cf2-4245-a87c-2083588b1892-kube-api-access-gv6rf\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.046965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.046960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvnp\" (UniqueName: \"kubernetes.io/projected/4537a962-0de7-4494-93af-46ce9698662b-kube-api-access-4jvnp\") pod \"seaweedfs-64568bcd49-6lk77\" (UID: \"4537a962-0de7-4494-93af-46ce9698662b\") " pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.047072 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.046993 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.147386 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.147356 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvnp\" (UniqueName: \"kubernetes.io/projected/4537a962-0de7-4494-93af-46ce9698662b-kube-api-access-4jvnp\") pod \"seaweedfs-64568bcd49-6lk77\" (UID: \"4537a962-0de7-4494-93af-46ce9698662b\") " pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.147528 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.147399 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.147528 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.147455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4537a962-0de7-4494-93af-46ce9698662b-data\") pod \"seaweedfs-64568bcd49-6lk77\" (UID: \"4537a962-0de7-4494-93af-46ce9698662b\") " pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.147528 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.147489 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv6rf\" (UniqueName: \"kubernetes.io/projected/2fdba674-1cf2-4245-a87c-2083588b1892-kube-api-access-gv6rf\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.147702 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:35:35.147551 2558 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 15 16:35:35.147702 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:35:35.147618 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert podName:2fdba674-1cf2-4245-a87c-2083588b1892 nodeName:}" failed. No retries permitted until 2026-09-15 16:35:35.647600866 +0000 UTC m=+532.416882676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert") pod "kserve-controller-manager-b589c8cc8-plx4c" (UID: "2fdba674-1cf2-4245-a87c-2083588b1892") : secret "kserve-webhook-server-cert" not found Sep 15 16:35:35.147840 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.147793 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4537a962-0de7-4494-93af-46ce9698662b-data\") pod \"seaweedfs-64568bcd49-6lk77\" (UID: \"4537a962-0de7-4494-93af-46ce9698662b\") " pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.158013 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.157989 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvnp\" (UniqueName: \"kubernetes.io/projected/4537a962-0de7-4494-93af-46ce9698662b-kube-api-access-4jvnp\") pod \"seaweedfs-64568bcd49-6lk77\" (UID: \"4537a962-0de7-4494-93af-46ce9698662b\") " pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.160929 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.160906 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv6rf\" (UniqueName: \"kubernetes.io/projected/2fdba674-1cf2-4245-a87c-2083588b1892-kube-api-access-gv6rf\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.327036 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.326946 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:35.593668 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.593644 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-6lk77"] Sep 15 16:35:35.595491 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:35:35.595461 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4537a962_0de7_4494_93af_46ce9698662b.slice/crio-f7f82b563655616bc57e9dae9b76d0de9cb0dc263520d6fa8bf3940aa25e15ba WatchSource:0}: Error finding container f7f82b563655616bc57e9dae9b76d0de9cb0dc263520d6fa8bf3940aa25e15ba: Status 404 returned error can't find the container with id f7f82b563655616bc57e9dae9b76d0de9cb0dc263520d6fa8bf3940aa25e15ba Sep 15 16:35:35.651953 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.651930 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.654209 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.654191 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert\") pod \"kserve-controller-manager-b589c8cc8-plx4c\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:35.885277 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:35.885155 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:36.154293 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:36.154261 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-plx4c"] Sep 15 16:35:36.161200 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:35:36.161169 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdba674_1cf2_4245_a87c_2083588b1892.slice/crio-70c4c5c8a23505cbdffee6059ec374cbf9037d7a1a266c66620a12de74601875 WatchSource:0}: Error finding container 70c4c5c8a23505cbdffee6059ec374cbf9037d7a1a266c66620a12de74601875: Status 404 returned error can't find the container with id 70c4c5c8a23505cbdffee6059ec374cbf9037d7a1a266c66620a12de74601875 Sep 15 16:35:36.210484 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:36.210452 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" event={"ID":"2fdba674-1cf2-4245-a87c-2083588b1892","Type":"ContainerStarted","Data":"70c4c5c8a23505cbdffee6059ec374cbf9037d7a1a266c66620a12de74601875"} Sep 15 16:35:36.211782 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:36.211753 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6lk77" event={"ID":"4537a962-0de7-4494-93af-46ce9698662b","Type":"ContainerStarted","Data":"f7f82b563655616bc57e9dae9b76d0de9cb0dc263520d6fa8bf3940aa25e15ba"} Sep 15 16:35:40.239104 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:40.239062 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" event={"ID":"2fdba674-1cf2-4245-a87c-2083588b1892","Type":"ContainerStarted","Data":"cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e"} Sep 15 16:35:40.239555 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:40.239158 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:35:40.240664 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:40.240641 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-6lk77" event={"ID":"4537a962-0de7-4494-93af-46ce9698662b","Type":"ContainerStarted","Data":"8860ba3f1d7b0d1b863ea334b7f0e0cf307da5086d6638230a8c2e4577c4a8ae"} Sep 15 16:35:40.240807 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:40.240692 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:35:40.254960 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:40.254917 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" podStartSLOduration=2.869675182 podStartE2EDuration="6.254898039s" totalImagesPullingTime="3.385222857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:40.254606841 +0000 UTC m=+537.023888669" watchObservedRunningTime="2026-09-15 16:35:40.254898039 +0000 UTC m=+537.024179866" Sep 15 16:35:40.271325 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:40.271277 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-6lk77" podStartSLOduration=2.254945257 podStartE2EDuration="6.271262982s" totalImagesPullingTime="4.016317725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:40.270159089 +0000 UTC m=+537.039440901" watchObservedRunningTime="2026-09-15 16:35:40.271262982 +0000 UTC m=+537.040544808" Sep 15 16:35:46.249311 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:35:46.249277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-6lk77" Sep 15 16:36:11.252369 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:11.252333 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:36:16.961965 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:16.961930 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-plx4c"] Sep 15 16:36:16.962425 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:16.962109 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" podUID="2fdba674-1cf2-4245-a87c-2083588b1892" containerName="manager" containerID="cri-o://cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e" gracePeriod=10 Sep 15 16:36:16.984518 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:16.984492 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-cnxhb"] Sep 15 16:36:16.986834 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:16.986815 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:16.994822 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:16.994802 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-cnxhb"] Sep 15 16:36:17.095910 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.095881 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4zb\" (UniqueName: \"kubernetes.io/projected/59039a73-dc66-4623-bcd7-bffbd42285fe-kube-api-access-8p4zb\") pod \"kserve-controller-manager-b589c8cc8-cnxhb\" (UID: \"59039a73-dc66-4623-bcd7-bffbd42285fe\") " pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.096057 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.095945 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59039a73-dc66-4623-bcd7-bffbd42285fe-cert\") pod \"kserve-controller-manager-b589c8cc8-cnxhb\" (UID: \"59039a73-dc66-4623-bcd7-bffbd42285fe\") " pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.197048 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.197022 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59039a73-dc66-4623-bcd7-bffbd42285fe-cert\") pod \"kserve-controller-manager-b589c8cc8-cnxhb\" (UID: \"59039a73-dc66-4623-bcd7-bffbd42285fe\") " pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.197177 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.197123 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4zb\" (UniqueName: \"kubernetes.io/projected/59039a73-dc66-4623-bcd7-bffbd42285fe-kube-api-access-8p4zb\") pod \"kserve-controller-manager-b589c8cc8-cnxhb\" (UID: \"59039a73-dc66-4623-bcd7-bffbd42285fe\") " pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.199315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.199288 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59039a73-dc66-4623-bcd7-bffbd42285fe-cert\") pod \"kserve-controller-manager-b589c8cc8-cnxhb\" (UID: \"59039a73-dc66-4623-bcd7-bffbd42285fe\") " pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.205678 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.205653 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4zb\" (UniqueName: \"kubernetes.io/projected/59039a73-dc66-4623-bcd7-bffbd42285fe-kube-api-access-8p4zb\") pod \"kserve-controller-manager-b589c8cc8-cnxhb\" (UID: \"59039a73-dc66-4623-bcd7-bffbd42285fe\") " pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.209370 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.209349 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:36:17.298196 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.298163 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv6rf\" (UniqueName: \"kubernetes.io/projected/2fdba674-1cf2-4245-a87c-2083588b1892-kube-api-access-gv6rf\") pod \"2fdba674-1cf2-4245-a87c-2083588b1892\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " Sep 15 16:36:17.298196 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.298193 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert\") pod \"2fdba674-1cf2-4245-a87c-2083588b1892\" (UID: \"2fdba674-1cf2-4245-a87c-2083588b1892\") " Sep 15 16:36:17.300206 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.300177 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert" (OuterVolumeSpecName: "cert") pod "2fdba674-1cf2-4245-a87c-2083588b1892" (UID: "2fdba674-1cf2-4245-a87c-2083588b1892"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:36:17.300314 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.300214 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdba674-1cf2-4245-a87c-2083588b1892-kube-api-access-gv6rf" (OuterVolumeSpecName: "kube-api-access-gv6rf") pod "2fdba674-1cf2-4245-a87c-2083588b1892" (UID: "2fdba674-1cf2-4245-a87c-2083588b1892"). InnerVolumeSpecName "kube-api-access-gv6rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:36:17.349540 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.349515 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:17.399230 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.399186 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv6rf\" (UniqueName: \"kubernetes.io/projected/2fdba674-1cf2-4245-a87c-2083588b1892-kube-api-access-gv6rf\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:36:17.399230 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.399212 2558 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fdba674-1cf2-4245-a87c-2083588b1892-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:36:17.405606 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.405578 2558 generic.go:356] "Generic (PLEG): container finished" podID="2fdba674-1cf2-4245-a87c-2083588b1892" containerID="cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e" exitCode=0 Sep 15 16:36:17.405742 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.405631 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" event={"ID":"2fdba674-1cf2-4245-a87c-2083588b1892","Type":"ContainerDied","Data":"cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e"} Sep 15 16:36:17.405742 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.405638 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" Sep 15 16:36:17.405742 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.405680 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-plx4c" event={"ID":"2fdba674-1cf2-4245-a87c-2083588b1892","Type":"ContainerDied","Data":"70c4c5c8a23505cbdffee6059ec374cbf9037d7a1a266c66620a12de74601875"} Sep 15 16:36:17.405742 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.405711 2558 scope.go:117] "RemoveContainer" containerID="cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e" Sep 15 16:36:17.416324 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.416302 2558 scope.go:117] "RemoveContainer" containerID="cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e" Sep 15 16:36:17.416776 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:17.416631 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e\": container with ID starting with cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e not found: ID does not exist" containerID="cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e" Sep 15 16:36:17.416776 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.416664 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e"} err="failed to get container status \"cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e\": rpc error: code = NotFound desc = could not find container \"cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e\": container with ID starting with cc6ba3d144147e5e720e44070abdf8a301b4942ebe3af9234f0400fd5648444e not found: ID does not exist" Sep 15 16:36:17.431179 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.431156 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-plx4c"] Sep 15 16:36:17.434712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.434686 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-plx4c"] Sep 15 16:36:17.617063 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.617040 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-cnxhb"] Sep 15 16:36:17.619250 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:36:17.619225 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59039a73_dc66_4623_bcd7_bffbd42285fe.slice/crio-d456bc0f2dede949c0dd6d6670d43eb865128020fe28a993c45107df1d9fa5c9 WatchSource:0}: Error finding container d456bc0f2dede949c0dd6d6670d43eb865128020fe28a993c45107df1d9fa5c9: Status 404 returned error can't find the container with id d456bc0f2dede949c0dd6d6670d43eb865128020fe28a993c45107df1d9fa5c9 Sep 15 16:36:17.726819 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:17.726787 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fdba674-1cf2-4245-a87c-2083588b1892" path="/var/lib/kubelet/pods/2fdba674-1cf2-4245-a87c-2083588b1892/volumes" Sep 15 16:36:18.413153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:18.413054 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" event={"ID":"59039a73-dc66-4623-bcd7-bffbd42285fe","Type":"ContainerStarted","Data":"d23f8aa68055d357cf259c55b21674091ccbaeca7a7c37308512835d50d8aa99"} Sep 15 16:36:18.413153 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:18.413116 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" event={"ID":"59039a73-dc66-4623-bcd7-bffbd42285fe","Type":"ContainerStarted","Data":"d456bc0f2dede949c0dd6d6670d43eb865128020fe28a993c45107df1d9fa5c9"} Sep 15 16:36:18.413607 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:18.413223 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:18.429426 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:18.429380 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" podStartSLOduration=1.966307636 podStartE2EDuration="2.429365349s" totalImagesPullingTime="463.057713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:18.428393311 +0000 UTC m=+575.197675138" watchObservedRunningTime="2026-09-15 16:36:18.429365349 +0000 UTC m=+575.198647240" Sep 15 16:36:43.643524 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:43.643501 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:36:43.643952 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:43.643501 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:36:49.424126 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:49.424090 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b589c8cc8-cnxhb" Sep 15 16:36:50.679191 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.679154 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-2qbmm"] Sep 15 16:36:50.679567 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.679551 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fdba674-1cf2-4245-a87c-2083588b1892" containerName="manager" Sep 15 16:36:50.679611 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.679570 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fdba674-1cf2-4245-a87c-2083588b1892" containerName="manager" Sep 15 16:36:50.679649 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.679628 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fdba674-1cf2-4245-a87c-2083588b1892" containerName="manager" Sep 15 16:36:50.683373 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.683354 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:50.686002 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.685985 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 15 16:36:50.686080 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.686019 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-fs67h\"" Sep 15 16:36:50.695920 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.695892 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2qbmm"] Sep 15 16:36:50.698100 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.698079 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-h2mhj"] Sep 15 16:36:50.701278 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.701260 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.704050 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.704025 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Sep 15 16:36:50.704142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.704046 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-hbp7r\"" Sep 15 16:36:50.704142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.704070 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Sep 15 16:36:50.711701 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.711681 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-h2mhj"] Sep 15 16:36:50.780639 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.780605 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-metrics-certs\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.780826 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.780663 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmlj7\" (UniqueName: \"kubernetes.io/projected/287aeffd-0769-4ad7-b5e2-0b81ec51135b-kube-api-access-mmlj7\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.780972 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.780948 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4tv\" (UniqueName: \"kubernetes.io/projected/943659f7-4b1e-4138-8556-d48d06bd80fa-kube-api-access-8h4tv\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:50.781076 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.781060 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943659f7-4b1e-4138-8556-d48d06bd80fa-tls-certs\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:50.781212 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.781192 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-cert\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.881801 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.881763 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943659f7-4b1e-4138-8556-d48d06bd80fa-tls-certs\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:50.881801 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.881807 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-cert\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.881839 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-metrics-certs\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.881859 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmlj7\" (UniqueName: \"kubernetes.io/projected/287aeffd-0769-4ad7-b5e2-0b81ec51135b-kube-api-access-mmlj7\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.881916 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4tv\" (UniqueName: \"kubernetes.io/projected/943659f7-4b1e-4138-8556-d48d06bd80fa-kube-api-access-8h4tv\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:50.881920 2558 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:50.881977 2558 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:50.881995 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943659f7-4b1e-4138-8556-d48d06bd80fa-tls-certs podName:943659f7-4b1e-4138-8556-d48d06bd80fa nodeName:}" failed. No retries permitted until 2026-09-15 16:36:51.381973601 +0000 UTC m=+608.151255411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/943659f7-4b1e-4138-8556-d48d06bd80fa-tls-certs") pod "model-serving-api-7b954ffc95-2qbmm" (UID: "943659f7-4b1e-4138-8556-d48d06bd80fa") : secret "model-serving-api-tls" not found Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:50.881926 2558 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 15 16:36:50.882040 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:50.882028 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-metrics-certs podName:287aeffd-0769-4ad7-b5e2-0b81ec51135b nodeName:}" failed. No retries permitted until 2026-09-15 16:36:51.382017331 +0000 UTC m=+608.151299155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-metrics-certs") pod "odh-model-controller-7b966d7984-h2mhj" (UID: "287aeffd-0769-4ad7-b5e2-0b81ec51135b") : secret "odh-model-controller-metrics-tls" not found Sep 15 16:36:50.882334 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:36:50.882057 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-cert podName:287aeffd-0769-4ad7-b5e2-0b81ec51135b nodeName:}" failed. No retries permitted until 2026-09-15 16:36:51.382039311 +0000 UTC m=+608.151321120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-cert") pod "odh-model-controller-7b966d7984-h2mhj" (UID: "287aeffd-0769-4ad7-b5e2-0b81ec51135b") : secret "odh-model-controller-webhook-cert" not found Sep 15 16:36:50.892610 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.892582 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmlj7\" (UniqueName: \"kubernetes.io/projected/287aeffd-0769-4ad7-b5e2-0b81ec51135b-kube-api-access-mmlj7\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:50.894960 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:50.894937 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4tv\" (UniqueName: \"kubernetes.io/projected/943659f7-4b1e-4138-8556-d48d06bd80fa-kube-api-access-8h4tv\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:51.386839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.386804 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943659f7-4b1e-4138-8556-d48d06bd80fa-tls-certs\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:51.386839 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.386843 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-cert\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:51.387106 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.386873 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-metrics-certs\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:51.389340 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.389315 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-cert\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:51.389454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.389404 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943659f7-4b1e-4138-8556-d48d06bd80fa-tls-certs\") pod \"model-serving-api-7b954ffc95-2qbmm\" (UID: \"943659f7-4b1e-4138-8556-d48d06bd80fa\") " pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:51.389454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.389450 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/287aeffd-0769-4ad7-b5e2-0b81ec51135b-metrics-certs\") pod \"odh-model-controller-7b966d7984-h2mhj\" (UID: \"287aeffd-0769-4ad7-b5e2-0b81ec51135b\") " pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:51.592815 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.592785 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:51.612630 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.612608 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:51.864295 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.864265 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-2qbmm"] Sep 15 16:36:51.867322 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:36:51.867283 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943659f7_4b1e_4138_8556_d48d06bd80fa.slice/crio-348f040b3e1709c8da5c8894ebc39d1c769cf4059799db4f7856999e8d8485e2 WatchSource:0}: Error finding container 348f040b3e1709c8da5c8894ebc39d1c769cf4059799db4f7856999e8d8485e2: Status 404 returned error can't find the container with id 348f040b3e1709c8da5c8894ebc39d1c769cf4059799db4f7856999e8d8485e2 Sep 15 16:36:51.887210 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:51.887189 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-h2mhj"] Sep 15 16:36:51.888521 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:36:51.888498 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287aeffd_0769_4ad7_b5e2_0b81ec51135b.slice/crio-e441208ac26d4d71f520d415be20448e844993cfd3d9ac3a1b1eefb45f30f54b WatchSource:0}: Error finding container e441208ac26d4d71f520d415be20448e844993cfd3d9ac3a1b1eefb45f30f54b: Status 404 returned error can't find the container with id e441208ac26d4d71f520d415be20448e844993cfd3d9ac3a1b1eefb45f30f54b Sep 15 16:36:52.575547 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:52.575510 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-h2mhj" event={"ID":"287aeffd-0769-4ad7-b5e2-0b81ec51135b","Type":"ContainerStarted","Data":"e441208ac26d4d71f520d415be20448e844993cfd3d9ac3a1b1eefb45f30f54b"} Sep 15 16:36:52.577286 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:52.577251 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2qbmm" event={"ID":"943659f7-4b1e-4138-8556-d48d06bd80fa","Type":"ContainerStarted","Data":"348f040b3e1709c8da5c8894ebc39d1c769cf4059799db4f7856999e8d8485e2"} Sep 15 16:36:55.594113 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:55.594072 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-h2mhj" event={"ID":"287aeffd-0769-4ad7-b5e2-0b81ec51135b","Type":"ContainerStarted","Data":"4b7dd2380c6f160257594e7668719ab70dcbf0ac3cdc61342635472474254c02"} Sep 15 16:36:55.594547 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:55.594176 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:36:55.595889 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:55.595859 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-2qbmm" event={"ID":"943659f7-4b1e-4138-8556-d48d06bd80fa","Type":"ContainerStarted","Data":"43f7c893d8119c9d70bc0d643586083e1c2500d09e334c76f6582e15b7b5d379"} Sep 15 16:36:55.596023 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:55.595915 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:36:55.610856 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:55.610811 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-h2mhj" podStartSLOduration=2.077875457 podStartE2EDuration="5.610797998s" totalImagesPullingTime="3.532922541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:55.609454594 +0000 UTC m=+612.378736422" watchObservedRunningTime="2026-09-15 16:36:55.610797998 +0000 UTC m=+612.380079825" Sep 15 16:36:55.625778 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:36:55.625676 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-2qbmm" podStartSLOduration=2.124642681 podStartE2EDuration="5.625660669s" totalImagesPullingTime="3.501017988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:55.625440675 +0000 UTC m=+612.394722516" watchObservedRunningTime="2026-09-15 16:36:55.625660669 +0000 UTC m=+612.394942495" Sep 15 16:37:06.604033 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:37:06.603944 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-h2mhj" Sep 15 16:37:06.605676 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:37:06.605646 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-2qbmm" Sep 15 16:38:38.595553 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.595477 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv"] Sep 15 16:38:38.598959 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.598942 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.601616 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.601593 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 15 16:38:38.601990 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.601970 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Sep 15 16:38:38.602315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.602296 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 15 16:38:38.602780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.602753 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 15 16:38:38.602881 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.602834 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 15 16:38:38.603093 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.603075 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Sep 15 16:38:38.603146 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.603079 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dpw6w\"" Sep 15 16:38:38.613290 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.613269 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv"] Sep 15 16:38:38.701689 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.701656 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.701876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.701694 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.701876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.701767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.701876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.701793 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.702099 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.701919 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btm2\" (UniqueName: \"kubernetes.io/projected/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-api-access-9btm2\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.803343 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.803309 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.803504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.803369 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btm2\" (UniqueName: \"kubernetes.io/projected/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-api-access-9btm2\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.803504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.803400 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.803504 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:38:38.803493 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found Sep 15 16:38:38.803504 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.803495 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.803713 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:38:38.803547 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls podName:4ba5fba9-907c-4e7b-ad4b-02d25aee6153 nodeName:}" failed. No retries permitted until 2026-09-15 16:38:39.30353017 +0000 UTC m=+716.072811990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls") pod "isvc-xgboost-graph-predictor-555fc69d56-6lsvv" (UID: "4ba5fba9-907c-4e7b-ad4b-02d25aee6153") : secret "isvc-xgboost-graph-predictor-serving-cert" not found Sep 15 16:38:38.803713 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.803579 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.803883 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.803861 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.804053 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.804036 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.804192 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.804176 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:38.816979 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:38.816958 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btm2\" (UniqueName: \"kubernetes.io/projected/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-api-access-9btm2\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:39.308837 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:39.308804 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:39.311360 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:39.311333 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-6lsvv\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:39.508809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:39.508772 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:38:39.795211 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:39.795133 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv"] Sep 15 16:38:39.798062 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:38:39.798032 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba5fba9_907c_4e7b_ad4b_02d25aee6153.slice/crio-519d3692c4ce6b2d0699d030836d55c73ec3deef9eba4e32d26d7ca3a9e73c3b WatchSource:0}: Error finding container 519d3692c4ce6b2d0699d030836d55c73ec3deef9eba4e32d26d7ca3a9e73c3b: Status 404 returned error can't find the container with id 519d3692c4ce6b2d0699d030836d55c73ec3deef9eba4e32d26d7ca3a9e73c3b Sep 15 16:38:40.074389 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:40.074296 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerStarted","Data":"519d3692c4ce6b2d0699d030836d55c73ec3deef9eba4e32d26d7ca3a9e73c3b"} Sep 15 16:38:44.099069 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:44.099030 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerStarted","Data":"c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d"} Sep 15 16:38:48.119003 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:48.118969 2558 generic.go:356] "Generic (PLEG): container finished" podID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerID="c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d" exitCode=0 Sep 15 16:38:48.119364 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:48.119041 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerDied","Data":"c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d"} Sep 15 16:38:48.120500 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:38:48.120481 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 16:39:06.218349 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:06.218310 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerStarted","Data":"54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8"} Sep 15 16:39:09.237319 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:09.237278 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerStarted","Data":"a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50"} Sep 15 16:39:09.237712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:09.237350 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:39:09.237712 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:09.237476 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:39:09.238500 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:09.238474 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:09.256059 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:09.256016 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podStartSLOduration=3.866097477 podStartE2EDuration="31.256002169s" totalImagesPullingTime="23.404460021s" totalInitContainerRuntime="3.985444671s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:39:09.253603308 +0000 UTC m=+746.022885136" watchObservedRunningTime="2026-09-15 16:39:09.256002169 +0000 UTC m=+746.025283996" Sep 15 16:39:10.242319 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:10.242278 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:15.246512 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:15.246480 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:39:15.247198 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:15.247160 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:25.248050 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:25.248011 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:35.247740 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:35.247695 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:45.247860 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:45.247816 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:55.247901 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:55.247860 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:39:58.280386 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.280356 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw"] Sep 15 16:39:58.282627 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.282612 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.285020 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.284991 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-19dbe-serving-cert\"" Sep 15 16:39:58.285126 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.284997 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-19dbe-kube-rbac-proxy-sar-config\"" Sep 15 16:39:58.292454 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.292427 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw"] Sep 15 16:39:58.384579 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.384549 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-proxy-tls\") pod \"switch-graph-19dbe-5c5f9dbf9c-465cw\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.384753 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.384586 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-openshift-service-ca-bundle\") pod \"switch-graph-19dbe-5c5f9dbf9c-465cw\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.486031 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.485998 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-proxy-tls\") pod \"switch-graph-19dbe-5c5f9dbf9c-465cw\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.486214 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.486039 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-openshift-service-ca-bundle\") pod \"switch-graph-19dbe-5c5f9dbf9c-465cw\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.486688 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.486669 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-openshift-service-ca-bundle\") pod \"switch-graph-19dbe-5c5f9dbf9c-465cw\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.488441 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.488423 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-proxy-tls\") pod \"switch-graph-19dbe-5c5f9dbf9c-465cw\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.593410 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.593330 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:39:58.861232 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:58.861212 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw"] Sep 15 16:39:58.863565 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:39:58.863538 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fd261a_33c2_4df8_bbd7_ed284b3ef5e1.slice/crio-3e8160c5b3b6ea0814e9a6ba162a87ccedb7892bd96cc510e535e1bd26428e76 WatchSource:0}: Error finding container 3e8160c5b3b6ea0814e9a6ba162a87ccedb7892bd96cc510e535e1bd26428e76: Status 404 returned error can't find the container with id 3e8160c5b3b6ea0814e9a6ba162a87ccedb7892bd96cc510e535e1bd26428e76 Sep 15 16:39:59.469222 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:39:59.469184 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" event={"ID":"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1","Type":"ContainerStarted","Data":"3e8160c5b3b6ea0814e9a6ba162a87ccedb7892bd96cc510e535e1bd26428e76"} Sep 15 16:40:01.480443 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:01.480410 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" event={"ID":"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1","Type":"ContainerStarted","Data":"bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153"} Sep 15 16:40:01.480863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:01.480470 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:40:01.496455 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:01.496409 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podStartSLOduration=1.296938702 podStartE2EDuration="3.496395002s" totalImagesPullingTime="2.1994563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:01.495500311 +0000 UTC m=+798.264782160" watchObservedRunningTime="2026-09-15 16:40:01.496395002 +0000 UTC m=+798.265676829" Sep 15 16:40:05.247118 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:05.247081 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Sep 15 16:40:07.490464 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:07.490386 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:40:08.458165 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:08.458132 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw"] Sep 15 16:40:08.458355 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:08.458312 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" containerID="cri-o://bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153" gracePeriod=30 Sep 15 16:40:12.489426 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:12.489381 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:40:15.248307 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:15.248276 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:40:17.488841 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:17.488799 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:40:22.489062 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:22.489026 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:40:22.489456 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:22.489135 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:40:27.488779 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:27.488742 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:40:32.489638 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:32.489598 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:40:37.489346 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:37.489300 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:40:38.614188 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.614158 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:40:38.656830 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.656795 2558 generic.go:356] "Generic (PLEG): container finished" podID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerID="bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153" exitCode=0 Sep 15 16:40:38.656971 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.656838 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" Sep 15 16:40:38.656971 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.656837 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" event={"ID":"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1","Type":"ContainerDied","Data":"bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153"} Sep 15 16:40:38.657064 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.656970 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw" event={"ID":"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1","Type":"ContainerDied","Data":"3e8160c5b3b6ea0814e9a6ba162a87ccedb7892bd96cc510e535e1bd26428e76"} Sep 15 16:40:38.657064 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.656999 2558 scope.go:117] "RemoveContainer" containerID="bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153" Sep 15 16:40:38.665180 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.665163 2558 scope.go:117] "RemoveContainer" containerID="bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153" Sep 15 16:40:38.665446 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:40:38.665430 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153\": container with ID starting with bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153 not found: ID does not exist" containerID="bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153" Sep 15 16:40:38.665498 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.665462 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153"} err="failed to get container status \"bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153\": rpc error: code = NotFound desc = could not find container \"bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153\": container with ID starting with bf96feb4de8b1a13a3a412870f5860d6f16d8b38b30450618798b2ea3e57f153 not found: ID does not exist" Sep 15 16:40:38.729164 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.729142 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-proxy-tls\") pod \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " Sep 15 16:40:38.729275 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.729221 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-openshift-service-ca-bundle\") pod \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\" (UID: \"48fd261a-33c2-4df8-bbd7-ed284b3ef5e1\") " Sep 15 16:40:38.729578 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.729558 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" (UID: "48fd261a-33c2-4df8-bbd7-ed284b3ef5e1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:40:38.731298 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.731276 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" (UID: "48fd261a-33c2-4df8-bbd7-ed284b3ef5e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:40:38.830415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.830392 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:40:38.830415 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.830413 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:40:38.978090 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.978056 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw"] Sep 15 16:40:38.982137 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:38.982057 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-19dbe-5c5f9dbf9c-465cw"] Sep 15 16:40:39.727498 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:39.727461 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" path="/var/lib/kubelet/pods/48fd261a-33c2-4df8-bbd7-ed284b3ef5e1/volumes" Sep 15 16:40:48.242561 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.242529 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs"] Sep 15 16:40:48.242984 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.242907 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" Sep 15 16:40:48.242984 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.242919 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" Sep 15 16:40:48.242984 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.242979 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fd261a-33c2-4df8-bbd7-ed284b3ef5e1" containerName="switch-graph-19dbe" Sep 15 16:40:48.247395 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.247375 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:48.249893 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.249873 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Sep 15 16:40:48.249893 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.249891 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Sep 15 16:40:48.253270 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.253246 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs"] Sep 15 16:40:48.413026 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.412995 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:48.413202 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.413131 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/401c9238-51bd-49da-80e5-ccb8f8a81f79-openshift-service-ca-bundle\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:48.514092 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.513999 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/401c9238-51bd-49da-80e5-ccb8f8a81f79-openshift-service-ca-bundle\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:48.514092 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.514045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:48.514332 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:40:48.514147 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Sep 15 16:40:48.514332 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:40:48.514225 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls podName:401c9238-51bd-49da-80e5-ccb8f8a81f79 nodeName:}" failed. No retries permitted until 2026-09-15 16:40:49.014203226 +0000 UTC m=+845.783485031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls") pod "model-chainer-566f7f6cdf-kxkvs" (UID: "401c9238-51bd-49da-80e5-ccb8f8a81f79") : secret "model-chainer-serving-cert" not found Sep 15 16:40:48.514705 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:48.514679 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/401c9238-51bd-49da-80e5-ccb8f8a81f79-openshift-service-ca-bundle\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:49.018635 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:49.018593 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:49.018836 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:40:49.018775 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Sep 15 16:40:49.018885 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:40:49.018841 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls podName:401c9238-51bd-49da-80e5-ccb8f8a81f79 nodeName:}" failed. No retries permitted until 2026-09-15 16:40:50.018827126 +0000 UTC m=+846.788108934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls") pod "model-chainer-566f7f6cdf-kxkvs" (UID: "401c9238-51bd-49da-80e5-ccb8f8a81f79") : secret "model-chainer-serving-cert" not found Sep 15 16:40:50.026798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.026758 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:50.029242 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.029220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") pod \"model-chainer-566f7f6cdf-kxkvs\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:50.059190 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.059169 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:50.327649 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.327627 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs"] Sep 15 16:40:50.329231 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:40:50.329201 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401c9238_51bd_49da_80e5_ccb8f8a81f79.slice/crio-6f6ed8a8b68b14f7714606d9fedce757000f35ec3527b1ebe07dfe2ac4fe0e2d WatchSource:0}: Error finding container 6f6ed8a8b68b14f7714606d9fedce757000f35ec3527b1ebe07dfe2ac4fe0e2d: Status 404 returned error can't find the container with id 6f6ed8a8b68b14f7714606d9fedce757000f35ec3527b1ebe07dfe2ac4fe0e2d Sep 15 16:40:50.715080 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.714985 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" event={"ID":"401c9238-51bd-49da-80e5-ccb8f8a81f79","Type":"ContainerStarted","Data":"5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7"} Sep 15 16:40:50.715080 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.715042 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" event={"ID":"401c9238-51bd-49da-80e5-ccb8f8a81f79","Type":"ContainerStarted","Data":"6f6ed8a8b68b14f7714606d9fedce757000f35ec3527b1ebe07dfe2ac4fe0e2d"} Sep 15 16:40:50.715307 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.715110 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:50.731315 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:50.731270 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podStartSLOduration=2.731257234 podStartE2EDuration="2.731257234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:50.73041561 +0000 UTC m=+847.499697451" watchObservedRunningTime="2026-09-15 16:40:50.731257234 +0000 UTC m=+847.500539102" Sep 15 16:40:56.725660 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:56.725631 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:40:58.331956 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.331923 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs"] Sep 15 16:40:58.332410 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.332136 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" containerID="cri-o://5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7" gracePeriod=30 Sep 15 16:40:58.500143 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.500105 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv"] Sep 15 16:40:58.500478 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.500442 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" containerID="cri-o://54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8" gracePeriod=30 Sep 15 16:40:58.500626 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.500493 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kube-rbac-proxy" containerID="cri-o://a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50" gracePeriod=30 Sep 15 16:40:58.756064 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.756035 2558 generic.go:356] "Generic (PLEG): container finished" podID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerID="a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50" exitCode=2 Sep 15 16:40:58.756064 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:40:58.756070 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerDied","Data":"a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50"} Sep 15 16:41:00.242784 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:00.242742 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Sep 15 16:41:01.728978 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:01.728934 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:41:02.143258 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.143231 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:41:02.232892 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.232862 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9btm2\" (UniqueName: \"kubernetes.io/projected/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-api-access-9btm2\") pod \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " Sep 15 16:41:02.232892 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.232895 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-rbac-proxy-sar-config\") pod \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " Sep 15 16:41:02.233134 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.232925 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls\") pod \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " Sep 15 16:41:02.233134 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.232963 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kserve-provision-location\") pod \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " Sep 15 16:41:02.233134 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.232986 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-cabundle-cert\") pod \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\" (UID: \"4ba5fba9-907c-4e7b-ad4b-02d25aee6153\") " Sep 15 16:41:02.233356 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.233317 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ba5fba9-907c-4e7b-ad4b-02d25aee6153" (UID: "4ba5fba9-907c-4e7b-ad4b-02d25aee6153"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:41:02.233480 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.233333 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ba5fba9-907c-4e7b-ad4b-02d25aee6153" (UID: "4ba5fba9-907c-4e7b-ad4b-02d25aee6153"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 16:41:02.233480 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.233383 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ba5fba9-907c-4e7b-ad4b-02d25aee6153" (UID: "4ba5fba9-907c-4e7b-ad4b-02d25aee6153"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:41:02.234966 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.234946 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ba5fba9-907c-4e7b-ad4b-02d25aee6153" (UID: "4ba5fba9-907c-4e7b-ad4b-02d25aee6153"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:41:02.235051 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.235035 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-api-access-9btm2" (OuterVolumeSpecName: "kube-api-access-9btm2") pod "4ba5fba9-907c-4e7b-ad4b-02d25aee6153" (UID: "4ba5fba9-907c-4e7b-ad4b-02d25aee6153"). InnerVolumeSpecName "kube-api-access-9btm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 16:41:02.333780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.333696 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btm2\" (UniqueName: \"kubernetes.io/projected/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-api-access-9btm2\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:02.333780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.333717 2558 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:02.333780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.333748 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:02.333780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.333756 2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-kserve-provision-location\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:02.333780 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.333765 2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba5fba9-907c-4e7b-ad4b-02d25aee6153-cabundle-cert\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:02.778123 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.778084 2558 generic.go:356] "Generic (PLEG): container finished" podID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerID="54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8" exitCode=0 Sep 15 16:41:02.778596 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.778151 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" Sep 15 16:41:02.778596 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.778176 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerDied","Data":"54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8"} Sep 15 16:41:02.778596 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.778225 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv" event={"ID":"4ba5fba9-907c-4e7b-ad4b-02d25aee6153","Type":"ContainerDied","Data":"519d3692c4ce6b2d0699d030836d55c73ec3deef9eba4e32d26d7ca3a9e73c3b"} Sep 15 16:41:02.778596 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.778251 2558 scope.go:117] "RemoveContainer" containerID="a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50" Sep 15 16:41:02.787362 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.787343 2558 scope.go:117] "RemoveContainer" containerID="54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8" Sep 15 16:41:02.794664 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.794644 2558 scope.go:117] "RemoveContainer" containerID="c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d" Sep 15 16:41:02.801580 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.801557 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv"] Sep 15 16:41:02.803906 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.803879 2558 scope.go:117] "RemoveContainer" containerID="a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50" Sep 15 16:41:02.804198 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:41:02.804177 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50\": container with ID starting with a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50 not found: ID does not exist" containerID="a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50" Sep 15 16:41:02.804293 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.804210 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50"} err="failed to get container status \"a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50\": rpc error: code = NotFound desc = could not find container \"a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50\": container with ID starting with a6b30e7390c430b56e942b3da0c4a5c6a87f8ac8a487ba3510720d5dc2311f50 not found: ID does not exist" Sep 15 16:41:02.804293 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.804234 2558 scope.go:117] "RemoveContainer" containerID="54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8" Sep 15 16:41:02.804514 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:41:02.804495 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8\": container with ID starting with 54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8 not found: ID does not exist" containerID="54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8" Sep 15 16:41:02.804581 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.804520 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8"} err="failed to get container status \"54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8\": rpc error: code = NotFound desc = could not find container \"54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8\": container with ID starting with 54ffe7953300a7b9e5f85e7be2271dbe3c61200ab15fd19f9fc3d48b103586c8 not found: ID does not exist" Sep 15 16:41:02.804581 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.804535 2558 scope.go:117] "RemoveContainer" containerID="c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d" Sep 15 16:41:02.804847 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:41:02.804826 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d\": container with ID starting with c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d not found: ID does not exist" containerID="c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d" Sep 15 16:41:02.804921 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.804852 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d"} err="failed to get container status \"c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d\": rpc error: code = NotFound desc = could not find container \"c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d\": container with ID starting with c7cd76247a2124a0fd8b50e133c256641681b3e799679086196cc572e42a620d not found: ID does not exist" Sep 15 16:41:02.805077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:02.805059 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-6lsvv"] Sep 15 16:41:03.728099 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:03.728068 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" path="/var/lib/kubelet/pods/4ba5fba9-907c-4e7b-ad4b-02d25aee6153/volumes" Sep 15 16:41:06.724576 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:06.724537 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:41:08.655817 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.655781 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86"] Sep 15 16:41:08.656255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656164 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="storage-initializer" Sep 15 16:41:08.656255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656175 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="storage-initializer" Sep 15 16:41:08.656255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656198 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kube-rbac-proxy" Sep 15 16:41:08.656255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656206 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kube-rbac-proxy" Sep 15 16:41:08.656255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656217 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" Sep 15 16:41:08.656255 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656223 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" Sep 15 16:41:08.656452 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656280 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kserve-container" Sep 15 16:41:08.656452 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.656291 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba5fba9-907c-4e7b-ad4b-02d25aee6153" containerName="kube-rbac-proxy" Sep 15 16:41:08.661191 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.661170 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:08.663509 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.663488 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-cf726-serving-cert\"" Sep 15 16:41:08.663615 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.663494 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-cf726-kube-rbac-proxy-sar-config\"" Sep 15 16:41:08.667706 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.667681 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86"] Sep 15 16:41:08.683318 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.683290 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:08.683433 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.683383 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-openshift-service-ca-bundle\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:08.784660 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.784620 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:08.784874 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.784710 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-openshift-service-ca-bundle\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:08.784874 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:41:08.784801 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-cf726-serving-cert: secret "switch-graph-cf726-serving-cert" not found Sep 15 16:41:08.784874 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:41:08.784866 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls podName:357c2f98-efd0-498f-99b9-6f5c2a3e5c56 nodeName:}" failed. No retries permitted until 2026-09-15 16:41:09.284850032 +0000 UTC m=+866.054131838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls") pod "switch-graph-cf726-66cb7b9f5f-c4q86" (UID: "357c2f98-efd0-498f-99b9-6f5c2a3e5c56") : secret "switch-graph-cf726-serving-cert" not found Sep 15 16:41:08.785549 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:08.785511 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-openshift-service-ca-bundle\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:09.289154 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:09.289095 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:09.291522 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:09.291502 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls\") pod \"switch-graph-cf726-66cb7b9f5f-c4q86\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:09.572501 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:09.572413 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:09.843351 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:09.843326 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86"] Sep 15 16:41:09.846030 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:41:09.846005 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357c2f98_efd0_498f_99b9_6f5c2a3e5c56.slice/crio-764075750b66fc6528b0c1fca9fbf4c668459389490919204f3d2b3a71285ab3 WatchSource:0}: Error finding container 764075750b66fc6528b0c1fca9fbf4c668459389490919204f3d2b3a71285ab3: Status 404 returned error can't find the container with id 764075750b66fc6528b0c1fca9fbf4c668459389490919204f3d2b3a71285ab3 Sep 15 16:41:10.817488 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:10.817452 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" event={"ID":"357c2f98-efd0-498f-99b9-6f5c2a3e5c56","Type":"ContainerStarted","Data":"d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072"} Sep 15 16:41:10.817656 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:10.817501 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" event={"ID":"357c2f98-efd0-498f-99b9-6f5c2a3e5c56","Type":"ContainerStarted","Data":"764075750b66fc6528b0c1fca9fbf4c668459389490919204f3d2b3a71285ab3"} Sep 15 16:41:10.817656 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:10.817532 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:10.833625 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:10.833545 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podStartSLOduration=2.833529489 podStartE2EDuration="2.833529489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:41:10.832636949 +0000 UTC m=+867.601918781" watchObservedRunningTime="2026-09-15 16:41:10.833529489 +0000 UTC m=+867.602811366" Sep 15 16:41:11.724696 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:11.724644 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:41:11.727258 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:11.727236 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:41:16.724079 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:16.724033 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:41:16.828715 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:16.828681 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:41:21.724880 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:21.724840 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:41:26.724293 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:26.724249 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:41:28.476086 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.476056 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:41:28.556908 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.556879 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/401c9238-51bd-49da-80e5-ccb8f8a81f79-openshift-service-ca-bundle\") pod \"401c9238-51bd-49da-80e5-ccb8f8a81f79\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " Sep 15 16:41:28.557084 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.556953 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") pod \"401c9238-51bd-49da-80e5-ccb8f8a81f79\" (UID: \"401c9238-51bd-49da-80e5-ccb8f8a81f79\") " Sep 15 16:41:28.557236 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.557210 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401c9238-51bd-49da-80e5-ccb8f8a81f79-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "401c9238-51bd-49da-80e5-ccb8f8a81f79" (UID: "401c9238-51bd-49da-80e5-ccb8f8a81f79"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:41:28.559037 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.559016 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "401c9238-51bd-49da-80e5-ccb8f8a81f79" (UID: "401c9238-51bd-49da-80e5-ccb8f8a81f79"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:41:28.657763 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.657656 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/401c9238-51bd-49da-80e5-ccb8f8a81f79-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:28.657763 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.657685 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401c9238-51bd-49da-80e5-ccb8f8a81f79-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:41:28.900379 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.900345 2558 generic.go:356] "Generic (PLEG): container finished" podID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerID="5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7" exitCode=0 Sep 15 16:41:28.900548 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.900400 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" Sep 15 16:41:28.900548 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.900424 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" event={"ID":"401c9238-51bd-49da-80e5-ccb8f8a81f79","Type":"ContainerDied","Data":"5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7"} Sep 15 16:41:28.900548 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.900475 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs" event={"ID":"401c9238-51bd-49da-80e5-ccb8f8a81f79","Type":"ContainerDied","Data":"6f6ed8a8b68b14f7714606d9fedce757000f35ec3527b1ebe07dfe2ac4fe0e2d"} Sep 15 16:41:28.900548 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.900499 2558 scope.go:117] "RemoveContainer" containerID="5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7" Sep 15 16:41:28.909603 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.909585 2558 scope.go:117] "RemoveContainer" containerID="5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7" Sep 15 16:41:28.909898 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:41:28.909878 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7\": container with ID starting with 5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7 not found: ID does not exist" containerID="5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7" Sep 15 16:41:28.909961 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.909906 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7"} err="failed to get container status \"5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7\": rpc error: code = NotFound desc = could not find container \"5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7\": container with ID starting with 5b77a6b0b783ad2280b55a4e294571ca5cdf7fadf560467313602db6074d22f7 not found: ID does not exist" Sep 15 16:41:28.921020 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.920999 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs"] Sep 15 16:41:28.925288 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:28.925269 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-566f7f6cdf-kxkvs"] Sep 15 16:41:29.727851 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:29.727819 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" path="/var/lib/kubelet/pods/401c9238-51bd-49da-80e5-ccb8f8a81f79/volumes" Sep 15 16:41:43.679842 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:43.679770 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:41:43.681142 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:43.681120 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:41:58.548619 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.548590 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv"] Sep 15 16:41:58.549013 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.548989 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" Sep 15 16:41:58.549013 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.549001 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" Sep 15 16:41:58.549083 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.549067 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="401c9238-51bd-49da-80e5-ccb8f8a81f79" containerName="model-chainer" Sep 15 16:41:58.553395 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.553375 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.555909 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.555886 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-daa19-serving-cert\"" Sep 15 16:41:58.556023 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.555917 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-daa19-kube-rbac-proxy-sar-config\"" Sep 15 16:41:58.562757 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.562736 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv"] Sep 15 16:41:58.706085 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.706048 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6d33604-a4c6-4674-9196-1b5817b79d26-openshift-service-ca-bundle\") pod \"sequence-graph-daa19-76989745dd-jw7xv\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.706249 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.706091 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d33604-a4c6-4674-9196-1b5817b79d26-proxy-tls\") pod \"sequence-graph-daa19-76989745dd-jw7xv\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.807566 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.807492 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6d33604-a4c6-4674-9196-1b5817b79d26-openshift-service-ca-bundle\") pod \"sequence-graph-daa19-76989745dd-jw7xv\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.807566 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.807529 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d33604-a4c6-4674-9196-1b5817b79d26-proxy-tls\") pod \"sequence-graph-daa19-76989745dd-jw7xv\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.808198 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.808172 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6d33604-a4c6-4674-9196-1b5817b79d26-openshift-service-ca-bundle\") pod \"sequence-graph-daa19-76989745dd-jw7xv\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.810085 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.810061 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d33604-a4c6-4674-9196-1b5817b79d26-proxy-tls\") pod \"sequence-graph-daa19-76989745dd-jw7xv\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:58.863558 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:58.863527 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:41:59.132390 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:41:59.132363 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv"] Sep 15 16:41:59.134870 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:41:59.134843 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d33604_a4c6_4674_9196_1b5817b79d26.slice/crio-f390955e2e42798bab8603dfe1c62b3a66b1db7be69ba4278f4816fe1293ccf5 WatchSource:0}: Error finding container f390955e2e42798bab8603dfe1c62b3a66b1db7be69ba4278f4816fe1293ccf5: Status 404 returned error can't find the container with id f390955e2e42798bab8603dfe1c62b3a66b1db7be69ba4278f4816fe1293ccf5 Sep 15 16:42:00.045244 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:42:00.045203 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" event={"ID":"c6d33604-a4c6-4674-9196-1b5817b79d26","Type":"ContainerStarted","Data":"6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf"} Sep 15 16:42:00.045813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:42:00.045258 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" event={"ID":"c6d33604-a4c6-4674-9196-1b5817b79d26","Type":"ContainerStarted","Data":"f390955e2e42798bab8603dfe1c62b3a66b1db7be69ba4278f4816fe1293ccf5"} Sep 15 16:42:00.045813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:42:00.045280 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:42:00.062676 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:42:00.062623 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podStartSLOduration=2.062606142 podStartE2EDuration="2.062606142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:00.060861769 +0000 UTC m=+916.830143596" watchObservedRunningTime="2026-09-15 16:42:00.062606142 +0000 UTC m=+916.831887966" Sep 15 16:42:06.056081 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:42:06.056054 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:46:43.709801 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:46:43.709700 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:46:43.711959 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:46:43.711934 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:49:23.359397 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:23.359314 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86"] Sep 15 16:49:23.361968 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:23.359584 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" containerID="cri-o://d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072" gracePeriod=30 Sep 15 16:49:26.827041 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:26.827005 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:49:31.826240 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:31.826197 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:49:36.826514 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:36.826467 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:49:36.826947 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:36.826581 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:49:41.826956 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:41.826915 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:49:46.826854 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:46.826817 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:49:51.826481 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:51.826445 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:49:53.518424 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.518400 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:49:53.663284 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.663199 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls\") pod \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " Sep 15 16:49:53.663284 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.663268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-openshift-service-ca-bundle\") pod \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\" (UID: \"357c2f98-efd0-498f-99b9-6f5c2a3e5c56\") " Sep 15 16:49:53.663647 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.663620 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "357c2f98-efd0-498f-99b9-6f5c2a3e5c56" (UID: "357c2f98-efd0-498f-99b9-6f5c2a3e5c56"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:49:53.665358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.665338 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "357c2f98-efd0-498f-99b9-6f5c2a3e5c56" (UID: "357c2f98-efd0-498f-99b9-6f5c2a3e5c56"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:49:53.764357 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.764331 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:49:53.764357 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:53.764356 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/357c2f98-efd0-498f-99b9-6f5c2a3e5c56-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:49:54.272451 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.272412 2558 generic.go:356] "Generic (PLEG): container finished" podID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerID="d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072" exitCode=0 Sep 15 16:49:54.272622 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.272465 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" Sep 15 16:49:54.272622 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.272499 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" event={"ID":"357c2f98-efd0-498f-99b9-6f5c2a3e5c56","Type":"ContainerDied","Data":"d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072"} Sep 15 16:49:54.272622 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.272578 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86" event={"ID":"357c2f98-efd0-498f-99b9-6f5c2a3e5c56","Type":"ContainerDied","Data":"764075750b66fc6528b0c1fca9fbf4c668459389490919204f3d2b3a71285ab3"} Sep 15 16:49:54.272622 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.272599 2558 scope.go:117] "RemoveContainer" containerID="d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072" Sep 15 16:49:54.281054 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.281042 2558 scope.go:117] "RemoveContainer" containerID="d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072" Sep 15 16:49:54.281290 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:49:54.281272 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072\": container with ID starting with d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072 not found: ID does not exist" containerID="d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072" Sep 15 16:49:54.281352 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.281297 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072"} err="failed to get container status \"d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072\": rpc error: code = NotFound desc = could not find container \"d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072\": container with ID starting with d40cbc63fe6ead6e4f0e86daf2cc6fb4e1febd134c53579b4d0369d4c9fa7072 not found: ID does not exist" Sep 15 16:49:54.287809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.287791 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86"] Sep 15 16:49:54.291359 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:54.291339 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-cf726-66cb7b9f5f-c4q86"] Sep 15 16:49:55.727169 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:49:55.727142 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" path="/var/lib/kubelet/pods/357c2f98-efd0-498f-99b9-6f5c2a3e5c56/volumes" Sep 15 16:50:13.300610 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:13.300577 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv"] Sep 15 16:50:13.301040 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:13.300837 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" containerID="cri-o://6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf" gracePeriod=30 Sep 15 16:50:16.054453 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:16.054408 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:21.054657 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:21.054607 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:23.585582 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.585549 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt"] Sep 15 16:50:23.585971 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.585935 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" Sep 15 16:50:23.585971 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.585948 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" Sep 15 16:50:23.586042 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.586014 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="357c2f98-efd0-498f-99b9-6f5c2a3e5c56" containerName="switch-graph-cf726" Sep 15 16:50:23.589055 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.589035 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:23.591375 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.591354 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fcea7-serving-cert\"" Sep 15 16:50:23.591482 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.591359 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fcea7-kube-rbac-proxy-sar-config\"" Sep 15 16:50:23.597140 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.597118 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt"] Sep 15 16:50:23.604741 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.604698 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:23.604846 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.604763 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85596251-0108-4627-9fce-3bd1d2b01162-openshift-service-ca-bundle\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:23.705471 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.705434 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:23.705471 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.705471 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85596251-0108-4627-9fce-3bd1d2b01162-openshift-service-ca-bundle\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:23.705672 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:50:23.705590 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-fcea7-serving-cert: secret "ensemble-graph-fcea7-serving-cert" not found Sep 15 16:50:23.705672 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:50:23.705668 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls podName:85596251-0108-4627-9fce-3bd1d2b01162 nodeName:}" failed. No retries permitted until 2026-09-15 16:50:24.205648988 +0000 UTC m=+1420.974930811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls") pod "ensemble-graph-fcea7-55b4968c6b-9nlgt" (UID: "85596251-0108-4627-9fce-3bd1d2b01162") : secret "ensemble-graph-fcea7-serving-cert" not found Sep 15 16:50:23.706224 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:23.706206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85596251-0108-4627-9fce-3bd1d2b01162-openshift-service-ca-bundle\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:24.209295 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:24.209254 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:24.211796 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:24.211763 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls\") pod \"ensemble-graph-fcea7-55b4968c6b-9nlgt\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:24.500841 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:24.500809 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:24.772874 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:24.772824 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt"] Sep 15 16:50:24.775289 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:50:24.775261 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85596251_0108_4627_9fce_3bd1d2b01162.slice/crio-3dbac55b4246721b15475ec400889b283211f5905f351609a6577b97d685d182 WatchSource:0}: Error finding container 3dbac55b4246721b15475ec400889b283211f5905f351609a6577b97d685d182: Status 404 returned error can't find the container with id 3dbac55b4246721b15475ec400889b283211f5905f351609a6577b97d685d182 Sep 15 16:50:25.423685 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:25.423645 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" event={"ID":"85596251-0108-4627-9fce-3bd1d2b01162","Type":"ContainerStarted","Data":"aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e"} Sep 15 16:50:25.423900 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:25.423708 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:25.423900 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:25.423751 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" event={"ID":"85596251-0108-4627-9fce-3bd1d2b01162","Type":"ContainerStarted","Data":"3dbac55b4246721b15475ec400889b283211f5905f351609a6577b97d685d182"} Sep 15 16:50:25.440013 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:25.439965 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podStartSLOduration=2.439950475 podStartE2EDuration="2.439950475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:25.438806416 +0000 UTC m=+1422.208088244" watchObservedRunningTime="2026-09-15 16:50:25.439950475 +0000 UTC m=+1422.209232302" Sep 15 16:50:26.054813 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:26.054778 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:26.055163 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:26.054874 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:50:31.054320 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:31.054280 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:31.434836 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:31.434739 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:33.646193 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:33.646117 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt"] Sep 15 16:50:33.646569 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:33.646341 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" containerID="cri-o://aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e" gracePeriod=30 Sep 15 16:50:36.054100 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:36.054015 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:36.432737 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:36.432642 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:41.054634 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:41.054592 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:41.432979 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:41.432899 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:43.439642 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.439617 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:50:43.509961 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.509926 2558 generic.go:356] "Generic (PLEG): container finished" podID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerID="6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf" exitCode=0 Sep 15 16:50:43.510112 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.509971 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" Sep 15 16:50:43.510112 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.510013 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" event={"ID":"c6d33604-a4c6-4674-9196-1b5817b79d26","Type":"ContainerDied","Data":"6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf"} Sep 15 16:50:43.510112 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.510068 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv" event={"ID":"c6d33604-a4c6-4674-9196-1b5817b79d26","Type":"ContainerDied","Data":"f390955e2e42798bab8603dfe1c62b3a66b1db7be69ba4278f4816fe1293ccf5"} Sep 15 16:50:43.510112 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.510092 2558 scope.go:117] "RemoveContainer" containerID="6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf" Sep 15 16:50:43.518178 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.518160 2558 scope.go:117] "RemoveContainer" containerID="6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf" Sep 15 16:50:43.518419 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:50:43.518400 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf\": container with ID starting with 6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf not found: ID does not exist" containerID="6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf" Sep 15 16:50:43.518480 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.518428 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf"} err="failed to get container status \"6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf\": rpc error: code = NotFound desc = could not find container \"6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf\": container with ID starting with 6471e8978dd2991d6509d81d4197510a212049202b7929aeed44622c11b48aaf not found: ID does not exist" Sep 15 16:50:43.571046 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.570998 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6d33604-a4c6-4674-9196-1b5817b79d26-openshift-service-ca-bundle\") pod \"c6d33604-a4c6-4674-9196-1b5817b79d26\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " Sep 15 16:50:43.571135 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.571083 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d33604-a4c6-4674-9196-1b5817b79d26-proxy-tls\") pod \"c6d33604-a4c6-4674-9196-1b5817b79d26\" (UID: \"c6d33604-a4c6-4674-9196-1b5817b79d26\") " Sep 15 16:50:43.571348 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.571323 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d33604-a4c6-4674-9196-1b5817b79d26-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c6d33604-a4c6-4674-9196-1b5817b79d26" (UID: "c6d33604-a4c6-4674-9196-1b5817b79d26"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:50:43.573163 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.573142 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d33604-a4c6-4674-9196-1b5817b79d26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6d33604-a4c6-4674-9196-1b5817b79d26" (UID: "c6d33604-a4c6-4674-9196-1b5817b79d26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:50:43.672077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.672049 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6d33604-a4c6-4674-9196-1b5817b79d26-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:50:43.672077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.672071 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6d33604-a4c6-4674-9196-1b5817b79d26-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:50:43.828011 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.827929 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv"] Sep 15 16:50:43.832596 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:43.832567 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-daa19-76989745dd-jw7xv"] Sep 15 16:50:45.728609 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:45.728574 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" path="/var/lib/kubelet/pods/c6d33604-a4c6-4674-9196-1b5817b79d26/volumes" Sep 15 16:50:46.432373 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:46.432326 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:46.432570 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:46.432436 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:50:51.432876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:51.432833 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:50:56.432196 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:50:56.432152 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:01.433047 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:01.433001 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:04.304226 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.304198 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:51:04.454745 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.454634 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls\") pod \"85596251-0108-4627-9fce-3bd1d2b01162\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " Sep 15 16:51:04.454920 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.454743 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85596251-0108-4627-9fce-3bd1d2b01162-openshift-service-ca-bundle\") pod \"85596251-0108-4627-9fce-3bd1d2b01162\" (UID: \"85596251-0108-4627-9fce-3bd1d2b01162\") " Sep 15 16:51:04.455074 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.455052 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85596251-0108-4627-9fce-3bd1d2b01162-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "85596251-0108-4627-9fce-3bd1d2b01162" (UID: "85596251-0108-4627-9fce-3bd1d2b01162"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:51:04.456710 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.456688 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85596251-0108-4627-9fce-3bd1d2b01162" (UID: "85596251-0108-4627-9fce-3bd1d2b01162"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:51:04.555938 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.555905 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85596251-0108-4627-9fce-3bd1d2b01162-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:51:04.555938 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.555933 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85596251-0108-4627-9fce-3bd1d2b01162-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:51:04.613585 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.613548 2558 generic.go:356] "Generic (PLEG): container finished" podID="85596251-0108-4627-9fce-3bd1d2b01162" containerID="aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e" exitCode=0 Sep 15 16:51:04.613809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.613598 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" Sep 15 16:51:04.613809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.613600 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" event={"ID":"85596251-0108-4627-9fce-3bd1d2b01162","Type":"ContainerDied","Data":"aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e"} Sep 15 16:51:04.613809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.613747 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt" event={"ID":"85596251-0108-4627-9fce-3bd1d2b01162","Type":"ContainerDied","Data":"3dbac55b4246721b15475ec400889b283211f5905f351609a6577b97d685d182"} Sep 15 16:51:04.613809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.613782 2558 scope.go:117] "RemoveContainer" containerID="aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e" Sep 15 16:51:04.623036 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.623019 2558 scope.go:117] "RemoveContainer" containerID="aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e" Sep 15 16:51:04.623313 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:51:04.623298 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e\": container with ID starting with aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e not found: ID does not exist" containerID="aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e" Sep 15 16:51:04.623369 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.623323 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e"} err="failed to get container status \"aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e\": rpc error: code = NotFound desc = could not find container \"aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e\": container with ID starting with aca988633362528a39d92d412b559733d584c6175f0f05331131cb98d451555e not found: ID does not exist" Sep 15 16:51:04.633973 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.633948 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt"] Sep 15 16:51:04.637293 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:04.637269 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fcea7-55b4968c6b-9nlgt"] Sep 15 16:51:05.728086 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:05.728051 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85596251-0108-4627-9fce-3bd1d2b01162" path="/var/lib/kubelet/pods/85596251-0108-4627-9fce-3bd1d2b01162/volumes" Sep 15 16:51:13.497262 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497225 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz"] Sep 15 16:51:13.497631 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497589 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" Sep 15 16:51:13.497631 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497600 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" Sep 15 16:51:13.497631 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497626 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" Sep 15 16:51:13.497631 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497632 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" Sep 15 16:51:13.497798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497695 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="85596251-0108-4627-9fce-3bd1d2b01162" containerName="ensemble-graph-fcea7" Sep 15 16:51:13.497798 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.497706 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d33604-a4c6-4674-9196-1b5817b79d26" containerName="sequence-graph-daa19" Sep 15 16:51:13.501902 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.501879 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.504184 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.504150 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 15 16:51:13.504184 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.504170 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-1d1fb-kube-rbac-proxy-sar-config\"" Sep 15 16:51:13.504390 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.504195 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dpw6w\"" Sep 15 16:51:13.504390 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.504171 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-1d1fb-serving-cert\"" Sep 15 16:51:13.507501 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.507479 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz"] Sep 15 16:51:13.529508 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.529487 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d353453d-f5cb-4f7d-8e34-28b595374dd3-openshift-service-ca-bundle\") pod \"sequence-graph-1d1fb-76459958d-swdvz\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.529618 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.529555 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d353453d-f5cb-4f7d-8e34-28b595374dd3-proxy-tls\") pod \"sequence-graph-1d1fb-76459958d-swdvz\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.630869 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.630831 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d353453d-f5cb-4f7d-8e34-28b595374dd3-proxy-tls\") pod \"sequence-graph-1d1fb-76459958d-swdvz\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.631033 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.630889 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d353453d-f5cb-4f7d-8e34-28b595374dd3-openshift-service-ca-bundle\") pod \"sequence-graph-1d1fb-76459958d-swdvz\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.631521 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.631496 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d353453d-f5cb-4f7d-8e34-28b595374dd3-openshift-service-ca-bundle\") pod \"sequence-graph-1d1fb-76459958d-swdvz\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.633212 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.633192 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d353453d-f5cb-4f7d-8e34-28b595374dd3-proxy-tls\") pod \"sequence-graph-1d1fb-76459958d-swdvz\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:13.812989 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:13.812885 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:14.083097 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:14.083014 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz"] Sep 15 16:51:14.086280 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:51:14.086252 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd353453d_f5cb_4f7d_8e34_28b595374dd3.slice/crio-3764e9984adb7a6e3fcd06b677b872d554117e859da37bcdca7d1255714b267f WatchSource:0}: Error finding container 3764e9984adb7a6e3fcd06b677b872d554117e859da37bcdca7d1255714b267f: Status 404 returned error can't find the container with id 3764e9984adb7a6e3fcd06b677b872d554117e859da37bcdca7d1255714b267f Sep 15 16:51:14.664778 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:14.664712 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" event={"ID":"d353453d-f5cb-4f7d-8e34-28b595374dd3","Type":"ContainerStarted","Data":"e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9"} Sep 15 16:51:14.665210 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:14.664795 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" event={"ID":"d353453d-f5cb-4f7d-8e34-28b595374dd3","Type":"ContainerStarted","Data":"3764e9984adb7a6e3fcd06b677b872d554117e859da37bcdca7d1255714b267f"} Sep 15 16:51:14.665210 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:14.664865 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:14.681087 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:14.681006 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podStartSLOduration=1.68098893 podStartE2EDuration="1.68098893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:51:14.680062123 +0000 UTC m=+1471.449343950" watchObservedRunningTime="2026-09-15 16:51:14.68098893 +0000 UTC m=+1471.450270757" Sep 15 16:51:20.674828 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:20.674798 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:23.590085 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:23.590049 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz"] Sep 15 16:51:23.590588 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:23.590248 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" containerID="cri-o://e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9" gracePeriod=30 Sep 15 16:51:25.673827 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:25.673788 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:30.673708 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:30.673665 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:35.674054 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:35.674015 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:35.674532 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:35.674135 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:40.673820 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:40.673779 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:43.741077 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.741047 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:51:43.742988 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.742970 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:51:43.879130 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.879097 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d"] Sep 15 16:51:43.884007 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.883977 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:43.886609 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.886586 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-0591a-kube-rbac-proxy-sar-config\"" Sep 15 16:51:43.886779 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.886589 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-0591a-serving-cert\"" Sep 15 16:51:43.890182 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:43.890109 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d"] Sep 15 16:51:44.001761 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.001650 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e9abf-5c92-4bfb-a541-bafc21a73a90-openshift-service-ca-bundle\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.001761 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.001698 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.103221 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.103177 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e9abf-5c92-4bfb-a541-bafc21a73a90-openshift-service-ca-bundle\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.103408 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.103231 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.103408 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:51:44.103341 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-0591a-serving-cert: secret "ensemble-graph-0591a-serving-cert" not found Sep 15 16:51:44.103408 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:51:44.103406 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls podName:943e9abf-5c92-4bfb-a541-bafc21a73a90 nodeName:}" failed. No retries permitted until 2026-09-15 16:51:44.603388688 +0000 UTC m=+1501.372670493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls") pod "ensemble-graph-0591a-587c6c6dd5-mlq7d" (UID: "943e9abf-5c92-4bfb-a541-bafc21a73a90") : secret "ensemble-graph-0591a-serving-cert" not found Sep 15 16:51:44.103864 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.103842 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e9abf-5c92-4bfb-a541-bafc21a73a90-openshift-service-ca-bundle\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.607864 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.607821 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.610358 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.610328 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls\") pod \"ensemble-graph-0591a-587c6c6dd5-mlq7d\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:44.794876 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:44.794828 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:45.063099 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:45.063066 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d"] Sep 15 16:51:45.069603 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:51:45.069567 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943e9abf_5c92_4bfb_a541_bafc21a73a90.slice/crio-bcf01cec3270d542d78669f8d94600d0a2c5483e4aa1612caa7e04c0d3dbde9b WatchSource:0}: Error finding container bcf01cec3270d542d78669f8d94600d0a2c5483e4aa1612caa7e04c0d3dbde9b: Status 404 returned error can't find the container with id bcf01cec3270d542d78669f8d94600d0a2c5483e4aa1612caa7e04c0d3dbde9b Sep 15 16:51:45.673452 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:45.673412 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:45.819696 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:45.819658 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" event={"ID":"943e9abf-5c92-4bfb-a541-bafc21a73a90","Type":"ContainerStarted","Data":"d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8"} Sep 15 16:51:45.820146 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:45.819717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" event={"ID":"943e9abf-5c92-4bfb-a541-bafc21a73a90","Type":"ContainerStarted","Data":"bcf01cec3270d542d78669f8d94600d0a2c5483e4aa1612caa7e04c0d3dbde9b"} Sep 15 16:51:45.820146 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:45.819776 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:45.835754 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:45.835678 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podStartSLOduration=2.83566477 podStartE2EDuration="2.83566477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:51:45.834476056 +0000 UTC m=+1502.603757885" watchObservedRunningTime="2026-09-15 16:51:45.83566477 +0000 UTC m=+1502.604946597" Sep 15 16:51:50.673950 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:50.673907 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 16:51:51.830419 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:51.830391 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 16:51:53.737011 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.736987 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:53.858460 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.858380 2558 generic.go:356] "Generic (PLEG): container finished" podID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerID="e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9" exitCode=0 Sep 15 16:51:53.858460 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.858428 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" Sep 15 16:51:53.858658 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.858465 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" event={"ID":"d353453d-f5cb-4f7d-8e34-28b595374dd3","Type":"ContainerDied","Data":"e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9"} Sep 15 16:51:53.858658 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.858523 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz" event={"ID":"d353453d-f5cb-4f7d-8e34-28b595374dd3","Type":"ContainerDied","Data":"3764e9984adb7a6e3fcd06b677b872d554117e859da37bcdca7d1255714b267f"} Sep 15 16:51:53.858658 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.858551 2558 scope.go:117] "RemoveContainer" containerID="e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9" Sep 15 16:51:53.867201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.867173 2558 scope.go:117] "RemoveContainer" containerID="e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9" Sep 15 16:51:53.867483 ip-10-0-142-196 kubenswrapper[2558]: E0915 16:51:53.867460 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9\": container with ID starting with e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9 not found: ID does not exist" containerID="e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9" Sep 15 16:51:53.867588 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.867500 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9"} err="failed to get container status \"e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9\": rpc error: code = NotFound desc = could not find container \"e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9\": container with ID starting with e97d7f8d917b02f4f747c6aaea2ea320fae8cf2b59d15ce6fd18a40000ca5ad9 not found: ID does not exist" Sep 15 16:51:53.889809 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.889788 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d353453d-f5cb-4f7d-8e34-28b595374dd3-proxy-tls\") pod \"d353453d-f5cb-4f7d-8e34-28b595374dd3\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " Sep 15 16:51:53.889930 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.889841 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d353453d-f5cb-4f7d-8e34-28b595374dd3-openshift-service-ca-bundle\") pod \"d353453d-f5cb-4f7d-8e34-28b595374dd3\" (UID: \"d353453d-f5cb-4f7d-8e34-28b595374dd3\") " Sep 15 16:51:53.890201 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.890177 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d353453d-f5cb-4f7d-8e34-28b595374dd3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d353453d-f5cb-4f7d-8e34-28b595374dd3" (UID: "d353453d-f5cb-4f7d-8e34-28b595374dd3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 16:51:53.891707 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.891681 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d353453d-f5cb-4f7d-8e34-28b595374dd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d353453d-f5cb-4f7d-8e34-28b595374dd3" (UID: "d353453d-f5cb-4f7d-8e34-28b595374dd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 16:51:53.990863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.990828 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d353453d-f5cb-4f7d-8e34-28b595374dd3-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:51:53.990863 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:53.990857 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d353453d-f5cb-4f7d-8e34-28b595374dd3-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 16:51:54.178934 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:54.178905 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz"] Sep 15 16:51:54.182694 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:54.182670 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-1d1fb-76459958d-swdvz"] Sep 15 16:51:55.727897 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:51:55.727862 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" path="/var/lib/kubelet/pods/d353453d-f5cb-4f7d-8e34-28b595374dd3/volumes" Sep 15 16:52:23.805995 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.805921 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw"] Sep 15 16:52:23.806431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.806342 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" Sep 15 16:52:23.806431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.806356 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" Sep 15 16:52:23.806431 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.806416 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d353453d-f5cb-4f7d-8e34-28b595374dd3" containerName="sequence-graph-1d1fb" Sep 15 16:52:23.809481 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.809465 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:23.811832 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.811806 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9d406-kube-rbac-proxy-sar-config\"" Sep 15 16:52:23.811938 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.811843 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-9d406-serving-cert\"" Sep 15 16:52:23.818413 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.818391 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw"] Sep 15 16:52:23.933000 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.932961 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/342ad092-1645-4149-87f4-3f21210e557d-proxy-tls\") pod \"sequence-graph-9d406-6b5656b658-2hfxw\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:23.933157 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:23.933015 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/342ad092-1645-4149-87f4-3f21210e557d-openshift-service-ca-bundle\") pod \"sequence-graph-9d406-6b5656b658-2hfxw\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:24.033600 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:24.033565 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/342ad092-1645-4149-87f4-3f21210e557d-proxy-tls\") pod \"sequence-graph-9d406-6b5656b658-2hfxw\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:24.033808 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:24.033610 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/342ad092-1645-4149-87f4-3f21210e557d-openshift-service-ca-bundle\") pod \"sequence-graph-9d406-6b5656b658-2hfxw\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:24.034302 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:24.034282 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/342ad092-1645-4149-87f4-3f21210e557d-openshift-service-ca-bundle\") pod \"sequence-graph-9d406-6b5656b658-2hfxw\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:24.036001 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:24.035982 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/342ad092-1645-4149-87f4-3f21210e557d-proxy-tls\") pod \"sequence-graph-9d406-6b5656b658-2hfxw\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:24.120686 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:24.120592 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:24.389656 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:24.389579 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw"] Sep 15 16:52:24.392628 ip-10-0-142-196 kubenswrapper[2558]: W0915 16:52:24.392599 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342ad092_1645_4149_87f4_3f21210e557d.slice/crio-3e95c4c8134b66dd380b9ff59d31812a910a8033e39de0e32f93bd0530831afc WatchSource:0}: Error finding container 3e95c4c8134b66dd380b9ff59d31812a910a8033e39de0e32f93bd0530831afc: Status 404 returned error can't find the container with id 3e95c4c8134b66dd380b9ff59d31812a910a8033e39de0e32f93bd0530831afc Sep 15 16:52:25.004483 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:25.004447 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" event={"ID":"342ad092-1645-4149-87f4-3f21210e557d","Type":"ContainerStarted","Data":"46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef"} Sep 15 16:52:25.004926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:25.004496 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" event={"ID":"342ad092-1645-4149-87f4-3f21210e557d","Type":"ContainerStarted","Data":"3e95c4c8134b66dd380b9ff59d31812a910a8033e39de0e32f93bd0530831afc"} Sep 15 16:52:25.004926 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:25.004598 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:52:25.021499 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:25.021449 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podStartSLOduration=2.021432334 podStartE2EDuration="2.021432334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:52:25.021112443 +0000 UTC m=+1541.790394269" watchObservedRunningTime="2026-09-15 16:52:25.021432334 +0000 UTC m=+1541.790714207" Sep 15 16:52:31.016066 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:52:31.016036 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 16:56:43.772763 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:56:43.772713 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:56:43.776039 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:56:43.776017 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 16:59:58.441394 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:59:58.441361 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d"] Sep 15 16:59:58.444057 ip-10-0-142-196 kubenswrapper[2558]: I0915 16:59:58.442172 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" containerID="cri-o://d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8" gracePeriod=30 Sep 15 17:00:01.828495 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:01.828443 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:06.828391 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:06.828350 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:11.828914 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:11.828876 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:11.829373 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:11.828974 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 17:00:16.828653 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:16.828616 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:21.827907 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:21.827866 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:26.827758 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:26.827706 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:28.596957 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.596926 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 17:00:28.705172 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.705086 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e9abf-5c92-4bfb-a541-bafc21a73a90-openshift-service-ca-bundle\") pod \"943e9abf-5c92-4bfb-a541-bafc21a73a90\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " Sep 15 17:00:28.705321 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.705194 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls\") pod \"943e9abf-5c92-4bfb-a541-bafc21a73a90\" (UID: \"943e9abf-5c92-4bfb-a541-bafc21a73a90\") " Sep 15 17:00:28.705475 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.705452 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943e9abf-5c92-4bfb-a541-bafc21a73a90-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "943e9abf-5c92-4bfb-a541-bafc21a73a90" (UID: "943e9abf-5c92-4bfb-a541-bafc21a73a90"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 17:00:28.707284 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.707258 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "943e9abf-5c92-4bfb-a541-bafc21a73a90" (UID: "943e9abf-5c92-4bfb-a541-bafc21a73a90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 17:00:28.805780 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.805745 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/943e9abf-5c92-4bfb-a541-bafc21a73a90-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:00:28.805780 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:28.805773 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/943e9abf-5c92-4bfb-a541-bafc21a73a90-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:00:29.241175 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.241142 2558 generic.go:356] "Generic (PLEG): container finished" podID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerID="d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8" exitCode=0 Sep 15 17:00:29.241386 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.241196 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" Sep 15 17:00:29.241386 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.241235 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" event={"ID":"943e9abf-5c92-4bfb-a541-bafc21a73a90","Type":"ContainerDied","Data":"d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8"} Sep 15 17:00:29.241386 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.241287 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d" event={"ID":"943e9abf-5c92-4bfb-a541-bafc21a73a90","Type":"ContainerDied","Data":"bcf01cec3270d542d78669f8d94600d0a2c5483e4aa1612caa7e04c0d3dbde9b"} Sep 15 17:00:29.241386 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.241308 2558 scope.go:117] "RemoveContainer" containerID="d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8" Sep 15 17:00:29.249861 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.249844 2558 scope.go:117] "RemoveContainer" containerID="d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8" Sep 15 17:00:29.250177 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:00:29.250158 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8\": container with ID starting with d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8 not found: ID does not exist" containerID="d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8" Sep 15 17:00:29.250244 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.250184 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8"} err="failed to get container status \"d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8\": rpc error: code = NotFound desc = could not find container \"d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8\": container with ID starting with d3c9c4a2e171bdc9e7c6b24c508e32036eb368f5a3f34062e6e4b2c19fd042b8 not found: ID does not exist" Sep 15 17:00:29.261456 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.261431 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d"] Sep 15 17:00:29.264767 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.264747 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-0591a-587c6c6dd5-mlq7d"] Sep 15 17:00:29.728018 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:29.727988 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" path="/var/lib/kubelet/pods/943e9abf-5c92-4bfb-a541-bafc21a73a90/volumes" Sep 15 17:00:38.540134 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:38.540099 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw"] Sep 15 17:00:38.540607 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:38.540302 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" containerID="cri-o://46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef" gracePeriod=30 Sep 15 17:00:41.012914 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:41.012877 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:46.013987 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:46.013942 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:51.013131 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:51.013094 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:51.013509 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:51.013190 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 17:00:56.013546 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:56.013510 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:00:58.778742 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.778698 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx"] Sep 15 17:00:58.779136 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.779067 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" Sep 15 17:00:58.779136 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.779078 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" Sep 15 17:00:58.779208 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.779150 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="943e9abf-5c92-4bfb-a541-bafc21a73a90" containerName="ensemble-graph-0591a" Sep 15 17:00:58.782009 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.781992 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:58.785020 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.784996 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-6a778-serving-cert\"" Sep 15 17:00:58.785246 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.785226 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-6a778-kube-rbac-proxy-sar-config\"" Sep 15 17:00:58.792837 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.792811 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx"] Sep 15 17:00:58.971677 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.971634 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ec9818f-5190-4235-9e07-0aee890adb04-proxy-tls\") pod \"splitter-graph-6a778-54754f5494-z6lzx\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:58.971890 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:58.971718 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec9818f-5190-4235-9e07-0aee890adb04-openshift-service-ca-bundle\") pod \"splitter-graph-6a778-54754f5494-z6lzx\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:59.072698 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.072627 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ec9818f-5190-4235-9e07-0aee890adb04-proxy-tls\") pod \"splitter-graph-6a778-54754f5494-z6lzx\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:59.072846 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.072697 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec9818f-5190-4235-9e07-0aee890adb04-openshift-service-ca-bundle\") pod \"splitter-graph-6a778-54754f5494-z6lzx\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:59.073362 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.073324 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec9818f-5190-4235-9e07-0aee890adb04-openshift-service-ca-bundle\") pod \"splitter-graph-6a778-54754f5494-z6lzx\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:59.075160 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.075134 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ec9818f-5190-4235-9e07-0aee890adb04-proxy-tls\") pod \"splitter-graph-6a778-54754f5494-z6lzx\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:59.093071 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.093044 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:00:59.361841 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.361816 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx"] Sep 15 17:00:59.364170 ip-10-0-142-196 kubenswrapper[2558]: W0915 17:00:59.364143 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec9818f_5190_4235_9e07_0aee890adb04.slice/crio-caefbe8ff1fa0fd38c568d7be43bf5bf0e8ea282adfab9e00cabbdd291602897 WatchSource:0}: Error finding container caefbe8ff1fa0fd38c568d7be43bf5bf0e8ea282adfab9e00cabbdd291602897: Status 404 returned error can't find the container with id caefbe8ff1fa0fd38c568d7be43bf5bf0e8ea282adfab9e00cabbdd291602897 Sep 15 17:00:59.387787 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:00:59.387760 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" event={"ID":"4ec9818f-5190-4235-9e07-0aee890adb04","Type":"ContainerStarted","Data":"caefbe8ff1fa0fd38c568d7be43bf5bf0e8ea282adfab9e00cabbdd291602897"} Sep 15 17:01:00.394344 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:00.394307 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" event={"ID":"4ec9818f-5190-4235-9e07-0aee890adb04","Type":"ContainerStarted","Data":"f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8"} Sep 15 17:01:00.394825 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:00.394441 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:01:00.410870 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:00.410822 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podStartSLOduration=2.41080675 podStartE2EDuration="2.41080675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:01:00.409202627 +0000 UTC m=+2057.178484456" watchObservedRunningTime="2026-09-15 17:01:00.41080675 +0000 UTC m=+2057.180088567" Sep 15 17:01:01.013888 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:01.013848 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:06.013011 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:06.012921 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:06.405660 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:06.405572 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:01:08.687590 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.687563 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 17:01:08.751987 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.751958 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/342ad092-1645-4149-87f4-3f21210e557d-openshift-service-ca-bundle\") pod \"342ad092-1645-4149-87f4-3f21210e557d\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " Sep 15 17:01:08.751987 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.751993 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/342ad092-1645-4149-87f4-3f21210e557d-proxy-tls\") pod \"342ad092-1645-4149-87f4-3f21210e557d\" (UID: \"342ad092-1645-4149-87f4-3f21210e557d\") " Sep 15 17:01:08.752345 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.752324 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342ad092-1645-4149-87f4-3f21210e557d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "342ad092-1645-4149-87f4-3f21210e557d" (UID: "342ad092-1645-4149-87f4-3f21210e557d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 17:01:08.754313 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.754286 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/342ad092-1645-4149-87f4-3f21210e557d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "342ad092-1645-4149-87f4-3f21210e557d" (UID: "342ad092-1645-4149-87f4-3f21210e557d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 17:01:08.820849 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.820815 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx"] Sep 15 17:01:08.821020 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.820998 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" containerID="cri-o://f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8" gracePeriod=30 Sep 15 17:01:08.853333 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.853306 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/342ad092-1645-4149-87f4-3f21210e557d-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:01:08.853333 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:08.853330 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/342ad092-1645-4149-87f4-3f21210e557d-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:01:09.438429 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.438393 2558 generic.go:356] "Generic (PLEG): container finished" podID="342ad092-1645-4149-87f4-3f21210e557d" containerID="46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef" exitCode=0 Sep 15 17:01:09.438615 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.438451 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" Sep 15 17:01:09.438615 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.438476 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" event={"ID":"342ad092-1645-4149-87f4-3f21210e557d","Type":"ContainerDied","Data":"46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef"} Sep 15 17:01:09.438615 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.438549 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw" event={"ID":"342ad092-1645-4149-87f4-3f21210e557d","Type":"ContainerDied","Data":"3e95c4c8134b66dd380b9ff59d31812a910a8033e39de0e32f93bd0530831afc"} Sep 15 17:01:09.438615 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.438570 2558 scope.go:117] "RemoveContainer" containerID="46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef" Sep 15 17:01:09.447127 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.447112 2558 scope.go:117] "RemoveContainer" containerID="46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef" Sep 15 17:01:09.447395 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:01:09.447378 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef\": container with ID starting with 46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef not found: ID does not exist" containerID="46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef" Sep 15 17:01:09.447440 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.447403 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef"} err="failed to get container status \"46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef\": rpc error: code = NotFound desc = could not find container \"46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef\": container with ID starting with 46d9a563e144eb76f27be3ca09b0f255b14b91e742b24af012df73067591f2ef not found: ID does not exist" Sep 15 17:01:09.458598 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.458570 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw"] Sep 15 17:01:09.461882 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.461864 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-9d406-6b5656b658-2hfxw"] Sep 15 17:01:09.728083 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:09.728053 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="342ad092-1645-4149-87f4-3f21210e557d" path="/var/lib/kubelet/pods/342ad092-1645-4149-87f4-3f21210e557d/volumes" Sep 15 17:01:11.403595 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:11.403553 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:16.403333 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:16.403296 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:21.403566 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:21.403521 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:21.404020 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:21.403680 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:01:26.403171 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:26.403130 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:31.403399 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:31.403360 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:36.403098 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:36.403057 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:01:38.769947 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.769909 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz"] Sep 15 17:01:38.770356 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.770289 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" Sep 15 17:01:38.770356 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.770302 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" Sep 15 17:01:38.770441 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.770361 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="342ad092-1645-4149-87f4-3f21210e557d" containerName="sequence-graph-9d406" Sep 15 17:01:38.773346 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.773327 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:38.775518 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.775497 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-969ee-kube-rbac-proxy-sar-config\"" Sep 15 17:01:38.775645 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.775500 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-969ee-serving-cert\"" Sep 15 17:01:38.781434 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.781411 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz"] Sep 15 17:01:38.906300 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.906263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:38.906456 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.906426 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c913c96-d21a-4649-a333-0c010a5a1945-openshift-service-ca-bundle\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:38.974607 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:38.974582 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:01:39.007227 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.007189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c913c96-d21a-4649-a333-0c010a5a1945-openshift-service-ca-bundle\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:39.007227 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.007228 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:39.007507 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:01:39.007330 2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-969ee-serving-cert: secret "switch-graph-969ee-serving-cert" not found Sep 15 17:01:39.007507 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:01:39.007385 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls podName:6c913c96-d21a-4649-a333-0c010a5a1945 nodeName:}" failed. No retries permitted until 2026-09-15 17:01:39.507368909 +0000 UTC m=+2096.276650715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls") pod "switch-graph-969ee-6dc8cf5c6c-t4wgz" (UID: "6c913c96-d21a-4649-a333-0c010a5a1945") : secret "switch-graph-969ee-serving-cert" not found Sep 15 17:01:39.007880 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.007858 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c913c96-d21a-4649-a333-0c010a5a1945-openshift-service-ca-bundle\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:39.108389 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.108298 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec9818f-5190-4235-9e07-0aee890adb04-openshift-service-ca-bundle\") pod \"4ec9818f-5190-4235-9e07-0aee890adb04\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " Sep 15 17:01:39.108543 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.108413 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ec9818f-5190-4235-9e07-0aee890adb04-proxy-tls\") pod \"4ec9818f-5190-4235-9e07-0aee890adb04\" (UID: \"4ec9818f-5190-4235-9e07-0aee890adb04\") " Sep 15 17:01:39.108681 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.108655 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec9818f-5190-4235-9e07-0aee890adb04-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4ec9818f-5190-4235-9e07-0aee890adb04" (UID: "4ec9818f-5190-4235-9e07-0aee890adb04"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 17:01:39.110648 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.110621 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ec9818f-5190-4235-9e07-0aee890adb04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ec9818f-5190-4235-9e07-0aee890adb04" (UID: "4ec9818f-5190-4235-9e07-0aee890adb04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 17:01:39.209324 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.209271 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ec9818f-5190-4235-9e07-0aee890adb04-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:01:39.209324 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.209316 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ec9818f-5190-4235-9e07-0aee890adb04-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:01:39.511855 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.511818 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:39.514357 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.514331 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls\") pod \"switch-graph-969ee-6dc8cf5c6c-t4wgz\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:39.580818 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.580782 2558 generic.go:356] "Generic (PLEG): container finished" podID="4ec9818f-5190-4235-9e07-0aee890adb04" containerID="f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8" exitCode=0 Sep 15 17:01:39.581023 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.580834 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" Sep 15 17:01:39.581023 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.580874 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" event={"ID":"4ec9818f-5190-4235-9e07-0aee890adb04","Type":"ContainerDied","Data":"f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8"} Sep 15 17:01:39.581023 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.580937 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx" event={"ID":"4ec9818f-5190-4235-9e07-0aee890adb04","Type":"ContainerDied","Data":"caefbe8ff1fa0fd38c568d7be43bf5bf0e8ea282adfab9e00cabbdd291602897"} Sep 15 17:01:39.581023 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.580971 2558 scope.go:117] "RemoveContainer" containerID="f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8" Sep 15 17:01:39.589671 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.589651 2558 scope.go:117] "RemoveContainer" containerID="f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8" Sep 15 17:01:39.590015 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:01:39.589993 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8\": container with ID starting with f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8 not found: ID does not exist" containerID="f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8" Sep 15 17:01:39.590083 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.590024 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8"} err="failed to get container status \"f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8\": rpc error: code = NotFound desc = could not find container \"f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8\": container with ID starting with f999ca561578fafbe95b205d96dfc5d863d60bde9c9d7615c94a44dd99019de8 not found: ID does not exist" Sep 15 17:01:39.602848 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.602815 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx"] Sep 15 17:01:39.606494 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.606467 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-6a778-54754f5494-z6lzx"] Sep 15 17:01:39.684321 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.684283 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:39.728186 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.728153 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" path="/var/lib/kubelet/pods/4ec9818f-5190-4235-9e07-0aee890adb04/volumes" Sep 15 17:01:39.955468 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:39.955445 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz"] Sep 15 17:01:39.957639 ip-10-0-142-196 kubenswrapper[2558]: W0915 17:01:39.957614 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c913c96_d21a_4649_a333_0c010a5a1945.slice/crio-f6bf7aa5618609aea1ef37a5e8391faa9b8f14f1314454b9a5af2ac26ad96f0f WatchSource:0}: Error finding container f6bf7aa5618609aea1ef37a5e8391faa9b8f14f1314454b9a5af2ac26ad96f0f: Status 404 returned error can't find the container with id f6bf7aa5618609aea1ef37a5e8391faa9b8f14f1314454b9a5af2ac26ad96f0f Sep 15 17:01:40.588145 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:40.588115 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" event={"ID":"6c913c96-d21a-4649-a333-0c010a5a1945","Type":"ContainerStarted","Data":"aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff"} Sep 15 17:01:40.588145 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:40.588163 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" event={"ID":"6c913c96-d21a-4649-a333-0c010a5a1945","Type":"ContainerStarted","Data":"f6bf7aa5618609aea1ef37a5e8391faa9b8f14f1314454b9a5af2ac26ad96f0f"} Sep 15 17:01:40.588361 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:40.588191 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:01:40.604137 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:40.604091 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podStartSLOduration=2.604074454 podStartE2EDuration="2.604074454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:01:40.603714664 +0000 UTC m=+2097.372996491" watchObservedRunningTime="2026-09-15 17:01:40.604074454 +0000 UTC m=+2097.373356282" Sep 15 17:01:43.803413 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:43.803385 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:01:43.807033 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:43.807009 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:01:46.598152 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:01:46.598120 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:02:19.072543 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.072508 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns"] Sep 15 17:02:19.073013 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.072914 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" Sep 15 17:02:19.073013 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.072926 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" Sep 15 17:02:19.073013 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.073003 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ec9818f-5190-4235-9e07-0aee890adb04" containerName="splitter-graph-6a778" Sep 15 17:02:19.075959 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.075942 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.078350 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.078318 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-1f299-kube-rbac-proxy-sar-config\"" Sep 15 17:02:19.078457 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.078320 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-1f299-serving-cert\"" Sep 15 17:02:19.086159 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.086134 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns"] Sep 15 17:02:19.145907 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.145875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-proxy-tls\") pod \"splitter-graph-1f299-85d87f69dc-7z9ns\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.146103 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.145931 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-openshift-service-ca-bundle\") pod \"splitter-graph-1f299-85d87f69dc-7z9ns\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.247020 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.246987 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-proxy-tls\") pod \"splitter-graph-1f299-85d87f69dc-7z9ns\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.247189 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.247045 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-openshift-service-ca-bundle\") pod \"splitter-graph-1f299-85d87f69dc-7z9ns\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.247611 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.247592 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-openshift-service-ca-bundle\") pod \"splitter-graph-1f299-85d87f69dc-7z9ns\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.249447 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.249427 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-proxy-tls\") pod \"splitter-graph-1f299-85d87f69dc-7z9ns\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.386341 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.386251 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.654217 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.654189 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns"] Sep 15 17:02:19.656147 ip-10-0-142-196 kubenswrapper[2558]: W0915 17:02:19.656113 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec15c26b_c74e_47e5_9eca_f9a873ef1db0.slice/crio-1c0dced8cbd78e1dd2119ac35203dc2ad265b03f8ce6c7fbd918782e82a3ac7a WatchSource:0}: Error finding container 1c0dced8cbd78e1dd2119ac35203dc2ad265b03f8ce6c7fbd918782e82a3ac7a: Status 404 returned error can't find the container with id 1c0dced8cbd78e1dd2119ac35203dc2ad265b03f8ce6c7fbd918782e82a3ac7a Sep 15 17:02:19.771327 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.771285 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" event={"ID":"ec15c26b-c74e-47e5-9eca-f9a873ef1db0","Type":"ContainerStarted","Data":"571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4"} Sep 15 17:02:19.771465 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.771346 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" event={"ID":"ec15c26b-c74e-47e5-9eca-f9a873ef1db0","Type":"ContainerStarted","Data":"1c0dced8cbd78e1dd2119ac35203dc2ad265b03f8ce6c7fbd918782e82a3ac7a"} Sep 15 17:02:19.771465 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.771368 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:02:19.787562 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:19.787520 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podStartSLOduration=0.787504077 podStartE2EDuration="787.504077ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:02:19.786115462 +0000 UTC m=+2136.555397285" watchObservedRunningTime="2026-09-15 17:02:19.787504077 +0000 UTC m=+2136.556785963" Sep 15 17:02:25.781476 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:02:25.781445 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:06:43.842528 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:06:43.842497 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:06:43.844871 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:06:43.844834 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:10:33.739941 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:33.739854 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns"] Sep 15 17:10:33.740467 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:33.740059 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" containerID="cri-o://571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4" gracePeriod=30 Sep 15 17:10:35.779610 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:35.779571 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:10:40.779942 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:40.779904 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:10:45.780100 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:45.780064 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:10:45.780493 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:45.780194 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:10:50.780110 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:50.780071 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:10:55.780619 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:10:55.780580 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:11:00.779516 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:00.779477 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:11:03.886588 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:03.886566 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:11:04.040209 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.040119 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-proxy-tls\") pod \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " Sep 15 17:11:04.040363 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.040233 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-openshift-service-ca-bundle\") pod \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\" (UID: \"ec15c26b-c74e-47e5-9eca-f9a873ef1db0\") " Sep 15 17:11:04.040596 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.040564 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ec15c26b-c74e-47e5-9eca-f9a873ef1db0" (UID: "ec15c26b-c74e-47e5-9eca-f9a873ef1db0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 17:11:04.042192 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.042170 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec15c26b-c74e-47e5-9eca-f9a873ef1db0" (UID: "ec15c26b-c74e-47e5-9eca-f9a873ef1db0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 17:11:04.141753 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.141692 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:11:04.141753 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.141745 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec15c26b-c74e-47e5-9eca-f9a873ef1db0-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:11:04.208483 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.208445 2558 generic.go:356] "Generic (PLEG): container finished" podID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerID="571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4" exitCode=0 Sep 15 17:11:04.208653 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.208491 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" Sep 15 17:11:04.208653 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.208490 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" event={"ID":"ec15c26b-c74e-47e5-9eca-f9a873ef1db0","Type":"ContainerDied","Data":"571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4"} Sep 15 17:11:04.208653 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.208622 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns" event={"ID":"ec15c26b-c74e-47e5-9eca-f9a873ef1db0","Type":"ContainerDied","Data":"1c0dced8cbd78e1dd2119ac35203dc2ad265b03f8ce6c7fbd918782e82a3ac7a"} Sep 15 17:11:04.208653 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.208651 2558 scope.go:117] "RemoveContainer" containerID="571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4" Sep 15 17:11:04.223377 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.223355 2558 scope.go:117] "RemoveContainer" containerID="571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4" Sep 15 17:11:04.223682 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:11:04.223664 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4\": container with ID starting with 571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4 not found: ID does not exist" containerID="571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4" Sep 15 17:11:04.223759 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.223689 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4"} err="failed to get container status \"571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4\": rpc error: code = NotFound desc = could not find container \"571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4\": container with ID starting with 571a939363bae68301c9439b217c8eb9d9d7eddd693179f9b0a2f9c300d900a4 not found: ID does not exist" Sep 15 17:11:04.231544 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.231511 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns"] Sep 15 17:11:04.235401 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:04.235376 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-1f299-85d87f69dc-7z9ns"] Sep 15 17:11:05.728330 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:05.728298 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" path="/var/lib/kubelet/pods/ec15c26b-c74e-47e5-9eca-f9a873ef1db0/volumes" Sep 15 17:11:43.872500 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:43.872474 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:11:43.875854 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:11:43.875826 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:16:43.901990 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:16:43.901916 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:16:43.906304 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:16:43.906281 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:17:57.912661 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:17:57.912621 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz"] Sep 15 17:17:57.913313 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:17:57.912908 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" containerID="cri-o://aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff" gracePeriod=30 Sep 15 17:18:01.596413 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:01.596370 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:18:06.596885 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:06.596846 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:18:11.597668 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:11.597630 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:18:11.598092 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:11.597770 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:18:15.156847 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:15.156816 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:15.959024 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:15.958997 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:16.597089 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:16.597044 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:18:16.757148 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:16.757109 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:17.556190 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:17.556154 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:18.335960 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:18.335935 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:19.126503 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:19.126470 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:19.913637 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:19.913588 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:20.714271 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:20.714242 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:21.489275 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:21.489244 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:21.597278 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:21.597236 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:18:22.290452 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:22.290417 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:23.100544 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:23.100512 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:23.889668 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:23.889634 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:24.674550 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:24.674519 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:25.448843 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:25.448811 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-969ee-6dc8cf5c6c-t4wgz_6c913c96-d21a-4649-a333-0c010a5a1945/switch-graph-969ee/0.log" Sep 15 17:18:26.596927 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:26.596891 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 17:18:28.053315 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.053292 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:18:28.200765 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.200629 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls\") pod \"6c913c96-d21a-4649-a333-0c010a5a1945\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " Sep 15 17:18:28.200765 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.200770 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c913c96-d21a-4649-a333-0c010a5a1945-openshift-service-ca-bundle\") pod \"6c913c96-d21a-4649-a333-0c010a5a1945\" (UID: \"6c913c96-d21a-4649-a333-0c010a5a1945\") " Sep 15 17:18:28.201104 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.201080 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c913c96-d21a-4649-a333-0c010a5a1945-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6c913c96-d21a-4649-a333-0c010a5a1945" (UID: "6c913c96-d21a-4649-a333-0c010a5a1945"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 17:18:28.202877 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.202856 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c913c96-d21a-4649-a333-0c010a5a1945" (UID: "6c913c96-d21a-4649-a333-0c010a5a1945"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 17:18:28.255168 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.255135 2558 generic.go:356] "Generic (PLEG): container finished" podID="6c913c96-d21a-4649-a333-0c010a5a1945" containerID="aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff" exitCode=0 Sep 15 17:18:28.255347 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.255182 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" Sep 15 17:18:28.255347 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.255190 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" event={"ID":"6c913c96-d21a-4649-a333-0c010a5a1945","Type":"ContainerDied","Data":"aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff"} Sep 15 17:18:28.255347 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.255243 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz" event={"ID":"6c913c96-d21a-4649-a333-0c010a5a1945","Type":"ContainerDied","Data":"f6bf7aa5618609aea1ef37a5e8391faa9b8f14f1314454b9a5af2ac26ad96f0f"} Sep 15 17:18:28.255347 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.255270 2558 scope.go:117] "RemoveContainer" containerID="aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff" Sep 15 17:18:28.263601 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.263583 2558 scope.go:117] "RemoveContainer" containerID="aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff" Sep 15 17:18:28.263886 ip-10-0-142-196 kubenswrapper[2558]: E0915 17:18:28.263862 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff\": container with ID starting with aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff not found: ID does not exist" containerID="aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff" Sep 15 17:18:28.263961 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.263898 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff"} err="failed to get container status \"aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff\": rpc error: code = NotFound desc = could not find container \"aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff\": container with ID starting with aa35b493724b60b64e650cb75c878a7cefa547ffde705d67f2aac02d653c43ff not found: ID does not exist" Sep 15 17:18:28.277860 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.277832 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz"] Sep 15 17:18:28.279464 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.279438 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-969ee-6dc8cf5c6c-t4wgz"] Sep 15 17:18:28.301690 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.301666 2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c913c96-d21a-4649-a333-0c010a5a1945-proxy-tls\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:18:28.301690 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:28.301689 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c913c96-d21a-4649-a333-0c010a5a1945-openshift-service-ca-bundle\") on node \"ip-10-0-142-196.ec2.internal\" DevicePath \"\"" Sep 15 17:18:29.727818 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:29.727719 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" path="/var/lib/kubelet/pods/6c913c96-d21a-4649-a333-0c010a5a1945/volumes" Sep 15 17:18:33.261409 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:33.261381 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-82z9t_6720930f-c883-40dc-bea0-11fc91a597c9/global-pull-secret-syncer/0.log" Sep 15 17:18:33.408321 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:33.408292 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gdxnf_31f363f1-7083-44b7-b0c9-8ee80b2ded87/konnectivity-agent/0.log" Sep 15 17:18:33.510672 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:33.510640 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-196.ec2.internal_c412a2301a21d6df633ce7962cca5842/haproxy/0.log" Sep 15 17:18:37.041331 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.041301 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/alertmanager/0.log" Sep 15 17:18:37.083025 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.082997 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/config-reloader/0.log" Sep 15 17:18:37.118141 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.118116 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/kube-rbac-proxy-web/0.log" Sep 15 17:18:37.157221 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.157195 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/kube-rbac-proxy/0.log" Sep 15 17:18:37.198124 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.198071 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/kube-rbac-proxy-metric/0.log" Sep 15 17:18:37.236070 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.236040 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/prom-label-proxy/0.log" Sep 15 17:18:37.270260 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.270238 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be9fd7d9-5578-45d4-8f18-972235c6bec9/init-config-reloader/0.log" Sep 15 17:18:37.319885 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.319810 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-rgx5w_6c5ce5cb-2b43-4718-99ce-22bc47aa26bf/cluster-monitoring-operator/0.log" Sep 15 17:18:37.579420 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.579350 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gj8cw_323937a7-e84e-4b71-af66-16f98e092a4a/node-exporter/0.log" Sep 15 17:18:37.604280 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.604259 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gj8cw_323937a7-e84e-4b71-af66-16f98e092a4a/kube-rbac-proxy/0.log" Sep 15 17:18:37.629766 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:37.629750 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gj8cw_323937a7-e84e-4b71-af66-16f98e092a4a/init-textfile/0.log" Sep 15 17:18:38.105306 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:38.105281 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db6f84fdb-w9zxc_2983aa89-38c9-4d25-8069-5d270cdf481f/telemeter-client/0.log" Sep 15 17:18:38.133641 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:38.133618 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db6f84fdb-w9zxc_2983aa89-38c9-4d25-8069-5d270cdf481f/reload/0.log" Sep 15 17:18:38.155940 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:38.155916 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-db6f84fdb-w9zxc_2983aa89-38c9-4d25-8069-5d270cdf481f/kube-rbac-proxy/0.log" Sep 15 17:18:39.712799 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:39.712774 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-6cfrt_9a8b302b-5c04-4f11-b592-2cb4a1adbfa6/networking-console-plugin/0.log" Sep 15 17:18:40.205597 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:40.205566 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/2.log" Sep 15 17:18:40.209670 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:40.209649 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-bfl42_4054a7ce-df3a-4ead-b7c9-032d28a0e1c7/console-operator/3.log" Sep 15 17:18:40.640977 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:40.640946 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9d6ff9cf4-q6txr_20539f6b-115b-497f-afea-fa6913c6c1be/console/0.log" Sep 15 17:18:40.675343 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:40.675317 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-bgmrq_37fb045f-d9a7-4fd5-a6de-4fc833ed4387/download-server/0.log" Sep 15 17:18:41.189437 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.189406 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-smmkq_12c4db27-9234-4d74-8b9f-cd83e1a835f0/volume-data-source-validator/0.log" Sep 15 17:18:41.865599 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.865558 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm"] Sep 15 17:18:41.866036 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.866017 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" Sep 15 17:18:41.866036 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.866037 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" Sep 15 17:18:41.866186 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.866070 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" Sep 15 17:18:41.866186 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.866079 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" Sep 15 17:18:41.866288 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.866201 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec15c26b-c74e-47e5-9eca-f9a873ef1db0" containerName="splitter-graph-1f299" Sep 15 17:18:41.866288 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.866217 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c913c96-d21a-4649-a333-0c010a5a1945" containerName="switch-graph-969ee" Sep 15 17:18:41.870563 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.870537 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:41.873015 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.872993 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4w98\"/\"kube-root-ca.crt\"" Sep 15 17:18:41.873942 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.873923 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t4w98\"/\"default-dockercfg-98xd7\"" Sep 15 17:18:41.874060 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.873950 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4w98\"/\"openshift-service-ca.crt\"" Sep 15 17:18:41.878875 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.878852 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm"] Sep 15 17:18:41.909250 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.909217 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszp5\" (UniqueName: \"kubernetes.io/projected/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-kube-api-access-wszp5\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:41.909432 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.909260 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-sys\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:41.909432 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.909304 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-lib-modules\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:41.909432 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.909380 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-podres\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:41.909432 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:41.909402 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-proc\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010541 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010500 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszp5\" (UniqueName: \"kubernetes.io/projected/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-kube-api-access-wszp5\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010541 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010543 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-sys\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010825 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010628 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-sys\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010825 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010664 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-lib-modules\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010825 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-podres\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010979 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-proc\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010979 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010864 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-lib-modules\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010979 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010922 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-proc\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.010979 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.010970 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-podres\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.018206 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.018179 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszp5\" (UniqueName: \"kubernetes.io/projected/fe077c45-89ab-4667-8ab1-5bfa7b4a66e5-kube-api-access-wszp5\") pod \"perf-node-gather-daemonset-h7mnm\" (UID: \"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5\") " pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.177066 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.176993 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zjwff_c9341790-00f2-4daf-884c-6c0157849da8/dns/0.log" Sep 15 17:18:42.181882 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.181865 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:42.205122 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.205099 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zjwff_c9341790-00f2-4daf-884c-6c0157849da8/kube-rbac-proxy/0.log" Sep 15 17:18:42.229589 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.229442 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jsdtg_0ed7aa9b-df82-4e1a-b35c-89087c2ced64/dns-node-resolver/0.log" Sep 15 17:18:42.453180 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.453156 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm"] Sep 15 17:18:42.455384 ip-10-0-142-196 kubenswrapper[2558]: W0915 17:18:42.455350 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe077c45_89ab_4667_8ab1_5bfa7b4a66e5.slice/crio-deaa049fa6efc79f6beaa4b65f798c0dee7676ec9a212a902e8c7a4e9f947460 WatchSource:0}: Error finding container deaa049fa6efc79f6beaa4b65f798c0dee7676ec9a212a902e8c7a4e9f947460: Status 404 returned error can't find the container with id deaa049fa6efc79f6beaa4b65f798c0dee7676ec9a212a902e8c7a4e9f947460 Sep 15 17:18:42.814688 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:42.814652 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ct27n_0cd19c25-3c74-418d-bab8-dc432ccc1e88/node-ca/0.log" Sep 15 17:18:43.326536 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:43.326504 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" event={"ID":"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5","Type":"ContainerStarted","Data":"e876e1c0edb18139194e2077e0bdb28398e5e9e608f891491c696414fd60ebe0"} Sep 15 17:18:43.326947 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:43.326555 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" event={"ID":"fe077c45-89ab-4667-8ab1-5bfa7b4a66e5","Type":"ContainerStarted","Data":"deaa049fa6efc79f6beaa4b65f798c0dee7676ec9a212a902e8c7a4e9f947460"} Sep 15 17:18:43.326947 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:43.326625 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:43.342753 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:43.342694 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" podStartSLOduration=2.342679958 podStartE2EDuration="2.342679958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 17:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:18:43.340584661 +0000 UTC m=+3120.109866490" watchObservedRunningTime="2026-09-15 17:18:43.342679958 +0000 UTC m=+3120.111961800" Sep 15 17:18:43.591295 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:43.591223 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-795444bb76-td5xf_5c23cb37-7ccf-4a86-9b6f-25b4bc588865/router/0.log" Sep 15 17:18:43.940498 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:43.940424 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k68km_c84043b1-49cb-4bd9-9dda-ac46fe09b427/serve-healthcheck-canary/0.log" Sep 15 17:18:44.308380 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:44.308345 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-n5jpm_1c5c1b79-6bb9-4718-adde-6280b2032021/insights-operator/0.log" Sep 15 17:18:44.309800 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:44.309784 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-n5jpm_1c5c1b79-6bb9-4718-adde-6280b2032021/insights-operator/1.log" Sep 15 17:18:44.406385 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:44.406351 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-br4bh_94cdbf57-fdeb-47f4-a90d-a2e636f0d156/kube-rbac-proxy/0.log" Sep 15 17:18:44.427265 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:44.427239 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-br4bh_94cdbf57-fdeb-47f4-a90d-a2e636f0d156/exporter/0.log" Sep 15 17:18:44.448351 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:44.448325 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-br4bh_94cdbf57-fdeb-47f4-a90d-a2e636f0d156/extractor/0.log" Sep 15 17:18:46.709783 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:46.709754 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-b589c8cc8-cnxhb_59039a73-dc66-4623-bcd7-bffbd42285fe/manager/0.log" Sep 15 17:18:46.761095 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:46.761065 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-2qbmm_943659f7-4b1e-4138-8556-d48d06bd80fa/server/0.log" Sep 15 17:18:46.854941 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:46.854910 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-h2mhj_287aeffd-0769-4ad7-b5e2-0b81ec51135b/manager/0.log" Sep 15 17:18:46.934773 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:46.934741 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-6lk77_4537a962-0de7-4494-93af-46ce9698662b/seaweedfs/0.log" Sep 15 17:18:49.340554 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:49.340524 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t4w98/perf-node-gather-daemonset-h7mnm" Sep 15 17:18:51.400207 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:51.400116 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-72tnw_68decd37-fefc-41d4-8a7b-ab6b40e1096b/kube-storage-version-migrator-operator/1.log" Sep 15 17:18:51.400947 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:51.400930 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-72tnw_68decd37-fefc-41d4-8a7b-ab6b40e1096b/kube-storage-version-migrator-operator/0.log" Sep 15 17:18:52.347766 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.347718 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/kube-multus-additional-cni-plugins/0.log" Sep 15 17:18:52.369507 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.369478 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/egress-router-binary-copy/0.log" Sep 15 17:18:52.390000 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.389981 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/cni-plugins/0.log" Sep 15 17:18:52.413890 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.413864 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/bond-cni-plugin/0.log" Sep 15 17:18:52.436155 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.436134 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/routeoverride-cni/0.log" Sep 15 17:18:52.457571 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.457523 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/whereabouts-cni-bincopy/0.log" Sep 15 17:18:52.478916 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.478897 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cj4xv_04db1029-90cb-4a75-a9c6-c0d1567cb33d/whereabouts-cni/0.log" Sep 15 17:18:52.912642 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:52.912610 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vj8vc_1f22ba54-c674-4533-a54a-7e7315f5562b/kube-multus/0.log" Sep 15 17:18:53.039375 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:53.039347 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wlfzm_1a2f1911-ce77-48aa-b3b8-c2199a13c65c/network-metrics-daemon/0.log" Sep 15 17:18:53.060052 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:53.060026 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wlfzm_1a2f1911-ce77-48aa-b3b8-c2199a13c65c/kube-rbac-proxy/0.log" Sep 15 17:18:54.169114 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.169079 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/ovn-controller/0.log" Sep 15 17:18:54.186383 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.186358 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/ovn-acl-logging/0.log" Sep 15 17:18:54.213897 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.213867 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/kube-rbac-proxy-node/0.log" Sep 15 17:18:54.234035 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.234010 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/kube-rbac-proxy-ovn-metrics/0.log" Sep 15 17:18:54.255195 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.255164 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/northd/0.log" Sep 15 17:18:54.281059 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.281033 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/nbdb/0.log" Sep 15 17:18:54.303185 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.303161 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/sbdb/0.log" Sep 15 17:18:54.344994 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:54.344970 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rwngp_f36bf4e7-e893-4d81-8285-66f26193a29a/ovnkube-controller/0.log" Sep 15 17:18:55.308942 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:55.308911 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-rtb2c_2f04dcb9-8302-4511-b762-ec05141ca84d/check-endpoints/0.log" Sep 15 17:18:55.360344 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:55.360305 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vgmpk_f1ac928e-2633-4ad0-9ab6-7c12f05c8610/network-check-target-container/0.log" Sep 15 17:18:56.266152 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:56.266114 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9ctfv_480f2cdd-804a-4d7f-b865-3ddd4da269c7/iptables-alerter/0.log" Sep 15 17:18:56.880139 ip-10-0-142-196 kubenswrapper[2558]: I0915 17:18:56.880113 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-crws5_c86ff085-8efe-41aa-8638-f4b36bf5d910/tuned/0.log"