Sep 09 23:06:19.028966 ip-10-0-140-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 23:06:19.028978 ip-10-0-140-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 23:06:19.028986 ip-10-0-140-142 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 23:06:19.029172 ip-10-0-140-142 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 23:06:29.247321 ip-10-0-140-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 23:06:29.247365 ip-10-0-140-142 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a02f36ca004e4b38922e57fa15c488d2 -- Sep 09 23:08:06.716621 ip-10-0-140-142 systemd[1]: Starting Kubernetes Kubelet... Sep 09 23:08:07.088747 ip-10-0-140-142 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:08:07.088747 ip-10-0-140-142 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 23:08:07.088747 ip-10-0-140-142 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:08:07.088747 ip-10-0-140-142 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 23:08:07.088747 ip-10-0-140-142 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 23:08:07.092255 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.092139 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 23:08:07.095610 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095587 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:08:07.095610 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095607 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:08:07.095610 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095611 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:08:07.095610 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095614 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:08:07.095610 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095617 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095622 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095625 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095628 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095632 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095652 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095655 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095658 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095661 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095664 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095666 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095669 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095672 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095675 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095678 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095680 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095683 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095688 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095691 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095694 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:08:07.095822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095698 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095700 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095704 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095707 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095710 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095712 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095717 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095721 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095724 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095727 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095729 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095732 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095734 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095736 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095740 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095743 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095745 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095748 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095756 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095759 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:08:07.096576 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095762 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095765 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095767 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095770 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095772 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095775 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095778 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095780 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095783 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095785 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095788 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095790 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095793 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095795 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095798 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095800 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095803 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095811 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095814 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095816 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:08:07.097335 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095819 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095822 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095825 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095827 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095830 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095832 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095836 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095838 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095842 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095846 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095849 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095857 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095861 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095863 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095865 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095868 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095871 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095873 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095876 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:08:07.098053 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095878 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095881 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095883 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095887 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095889 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095892 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095894 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095897 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095899 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095902 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095905 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095907 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095910 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095913 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095915 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.095918 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096717 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096725 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096728 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:08:07.098767 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096732 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096734 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096737 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096740 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096744 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096746 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096755 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096757 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096760 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096762 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096765 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096768 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096770 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096773 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096775 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096778 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096781 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096784 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096787 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:08:07.099524 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096789 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096791 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096794 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096797 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096799 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096802 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096804 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096807 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096809 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096812 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096814 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096817 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096820 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096823 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096826 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096829 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096831 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096834 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096837 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096839 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096847 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:08:07.100266 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096850 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096853 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096856 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096858 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096860 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096863 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096866 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096868 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096870 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096873 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096875 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096878 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096881 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096885 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096888 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096890 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096893 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096896 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096898 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:08:07.101071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096901 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096905 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096909 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096912 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096915 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096918 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096922 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096926 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096929 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096932 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096935 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096939 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096941 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096944 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096947 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096950 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096953 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096955 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096959 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:08:07.101786 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096963 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096965 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096968 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096970 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096973 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096975 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096978 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096980 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096983 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096985 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096988 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096990 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096993 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096995 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.096998 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097000 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097003 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097006 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097502 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097507 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:08:07.102519 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097510 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097513 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097516 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097519 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097521 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097524 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097532 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097535 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097538 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097540 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097543 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097547 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097551 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097554 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097557 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097560 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097563 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097565 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097568 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:08:07.103267 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097571 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097573 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097576 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097579 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097581 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097584 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097587 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097589 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097592 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097595 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097598 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097600 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097603 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097605 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097608 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097610 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097613 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097615 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097618 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097621 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:08:07.103970 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097629 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097632 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097635 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097637 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097640 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097642 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097645 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097647 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097650 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097652 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097655 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097657 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097660 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097665 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097668 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097670 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097673 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097675 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097678 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097680 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:08:07.104736 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097683 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097686 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097688 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097690 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097693 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097696 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097698 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097701 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097704 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097706 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097709 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097711 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097714 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097723 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097725 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097728 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097731 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097733 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097735 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097738 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:08:07.105547 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097742 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097745 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097749 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097751 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097754 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097758 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097761 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097764 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097766 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097769 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097771 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097774 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097776 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097779 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097781 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097784 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097786 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.097790 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097892 2571 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097931 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 23:08:07.106304 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097940 2571 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097945 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097949 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097952 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097959 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097963 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097973 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097976 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097980 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097983 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097987 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097990 2571 flags.go:64] FLAG: --cgroup-root="" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097993 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097996 2571 flags.go:64] FLAG: --client-ca-file="" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.097999 2571 flags.go:64] FLAG: --cloud-provider="external" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.098002 2571 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099733 2571 flags.go:64] FLAG: --cluster-domain="" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099743 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099747 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099751 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099755 2571 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099769 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099772 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 23:08:07.107078 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099775 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099779 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099782 2571 flags.go:64] FLAG: --contention-profiling="false" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099786 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099789 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099792 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099795 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099800 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099804 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099807 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099810 2571 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099813 2571 flags.go:64] FLAG: --enable-server="true" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099816 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099827 2571 flags.go:64] FLAG: --event-burst="100" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099831 2571 flags.go:64] FLAG: --event-qps="50" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099834 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099837 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099840 2571 flags.go:64] FLAG: --eviction-hard="" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099844 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099847 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099850 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099853 2571 flags.go:64] FLAG: --eviction-soft="" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099855 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099858 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099863 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 23:08:07.107936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099866 2571 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099869 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099872 2571 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099875 2571 flags.go:64] FLAG: --feature-gates="" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099879 2571 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099882 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099885 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099888 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099892 2571 flags.go:64] FLAG: --healthz-port="10248" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099895 2571 flags.go:64] FLAG: --help="false" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099898 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099901 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099904 2571 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099907 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099911 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099915 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099918 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099921 2571 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099923 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099926 2571 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099929 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099932 2571 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099935 2571 flags.go:64] FLAG: --kube-reserved="" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099938 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 23:08:07.108903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099940 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099944 2571 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099946 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099949 2571 flags.go:64] FLAG: --lock-file="" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099952 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099955 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099958 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099966 2571 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099971 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099974 2571 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099977 2571 flags.go:64] FLAG: --logging-format="text" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099980 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099983 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099986 2571 flags.go:64] FLAG: --manifest-url="" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099989 2571 flags.go:64] FLAG: --manifest-url-header="" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099994 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.099997 2571 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100002 2571 flags.go:64] FLAG: --max-pods="110" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100005 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100008 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100011 2571 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100014 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100017 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100020 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100023 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 23:08:07.109831 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100033 2571 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100036 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100046 2571 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100049 2571 flags.go:64] FLAG: --pod-cidr="" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100052 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100058 2571 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100061 2571 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100064 2571 flags.go:64] FLAG: --pods-per-core="0" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100067 2571 flags.go:64] FLAG: --port="10250" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100070 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100073 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-069cb8774512e526f" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100076 2571 flags.go:64] FLAG: --qos-reserved="" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100079 2571 flags.go:64] FLAG: --read-only-port="10255" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100082 2571 flags.go:64] FLAG: --register-node="true" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100085 2571 flags.go:64] FLAG: --register-with-taints="" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100092 2571 flags.go:64] FLAG: --registry-burst="10" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100095 2571 flags.go:64] FLAG: --registry-qps="5" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100098 2571 flags.go:64] FLAG: --reserved-cpus="" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100101 2571 flags.go:64] FLAG: --reserved-memory="" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100104 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100107 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100110 2571 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100113 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100116 2571 flags.go:64] FLAG: --runonce="false" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100119 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 23:08:07.110801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100122 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100125 2571 flags.go:64] FLAG: --seccomp-default="false" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100128 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100131 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100135 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100138 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100140 2571 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100143 2571 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100146 2571 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100149 2571 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100157 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100160 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100163 2571 flags.go:64] FLAG: --system-cgroups="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100166 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100171 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100174 2571 flags.go:64] FLAG: --tls-cert-file="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100177 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100182 2571 flags.go:64] FLAG: --tls-min-version="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100184 2571 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100187 2571 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100190 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100192 2571 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100199 2571 flags.go:64] FLAG: --v="2" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100203 2571 flags.go:64] FLAG: --version="false" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100212 2571 flags.go:64] FLAG: --vmodule="" Sep 09 23:08:07.111747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100215 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.100219 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100408 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100415 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100419 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100422 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100425 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100428 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100430 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100433 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100436 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100439 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100441 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100444 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100447 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100449 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100452 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100454 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100464 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100466 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100469 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:08:07.112733 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100471 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100474 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100477 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100479 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100482 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100485 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100496 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100500 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100505 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100508 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100511 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100514 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100518 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100522 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100525 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100527 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100530 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100533 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100535 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:08:07.113548 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100538 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100541 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100543 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100546 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100549 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100551 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100554 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100556 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100559 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100562 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100564 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100573 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100576 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100579 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100581 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100584 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100586 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100589 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100592 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100594 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:08:07.114365 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100597 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100601 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100604 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100607 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100611 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100614 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100617 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100620 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100623 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100625 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100628 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100630 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100633 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100635 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100638 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100640 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100643 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100645 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100648 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:08:07.115121 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100650 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100653 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100655 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100658 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100660 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100668 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100671 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100674 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100677 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100679 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100683 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100686 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100689 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100692 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100698 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100700 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100703 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100705 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100708 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:08:07.115838 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100711 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100713 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.100716 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.101222 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.108127 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.108147 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.108850 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.108859 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108923 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108930 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108935 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108938 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108941 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108943 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:08:07.116586 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108946 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108949 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108952 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108955 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108957 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108960 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108962 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108965 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108969 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108971 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108974 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108976 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108979 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108982 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108984 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108987 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108989 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108993 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.108997 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:08:07.117128 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109000 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109003 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109006 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109010 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109014 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109017 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109020 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109022 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109025 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109028 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109030 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109033 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109035 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109038 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109040 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109043 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109047 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109050 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109054 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:08:07.117929 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109057 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109060 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109063 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109067 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109070 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109074 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109076 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109079 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109082 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109084 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109087 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109090 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109092 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109095 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109098 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109100 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109104 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109107 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109113 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:08:07.118671 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109116 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109119 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109122 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109124 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109127 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109130 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109132 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109135 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109138 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109140 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109143 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109146 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109149 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109152 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109154 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109157 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109160 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109162 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109165 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109167 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:08:07.119438 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109170 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109173 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109176 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109179 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109181 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109184 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109186 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109189 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109192 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109195 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109198 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109201 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109204 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109206 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109209 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:08:07.120231 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109211 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.109217 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109372 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109379 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109382 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109385 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109388 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109391 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109395 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109398 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109400 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109403 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109406 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109409 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109411 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 23:08:07.120822 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109414 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109417 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109419 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109422 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109425 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109427 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109430 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109432 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109435 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109437 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109440 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109443 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109446 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109449 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109451 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109455 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109458 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109461 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109463 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109466 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 23:08:07.121409 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109469 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109471 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109474 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109476 2571 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109479 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109481 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109484 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109487 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109489 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109492 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109494 2571 feature_gate.go:328] unrecognized feature gate: Example Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109497 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109500 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109502 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109506 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109511 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109514 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109516 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109519 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 23:08:07.122162 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109521 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109524 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109527 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109529 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109532 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109535 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109537 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109540 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109542 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109545 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109548 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109550 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109553 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109556 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109558 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109561 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109563 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109566 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109570 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 23:08:07.122874 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109573 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109576 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109579 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109582 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109584 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109587 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109590 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109593 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109596 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109598 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109602 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109605 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109608 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109611 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109613 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109616 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109618 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109621 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109624 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 23:08:07.123597 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109627 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109629 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109632 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109635 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109638 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109641 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109644 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109646 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:07.109649 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.109654 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.109826 2571 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.111830 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.112619 2571 server.go:1019] "Starting client certificate rotation" Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.112743 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:08:07.124425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.112775 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 23:08:07.124968 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.114767 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:08:07.124968 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.115962 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 23:08:07.129052 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.129029 2571 log.go:25] "Validated CRI v1 runtime API" Sep 09 23:08:07.131969 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.131947 2571 log.go:25] "Validated CRI v1 image API" Sep 09 23:08:07.133001 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.132982 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 23:08:07.138148 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.138123 2571 fs.go:135] Filesystem UUIDs: map[577b8d75-bb12-414d-be7f-375faf55fa60:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7c1a3f75-838e-415b-a10f-2e7fecdc3c74:/dev/nvme0n1p4] Sep 09 23:08:07.138148 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.138145 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 23:08:07.144258 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.144118 2571 manager.go:216] Machine: {Timestamp:2026-09-09 23:08:07.143021616 +0000 UTC m=+0.315309693 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100891 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fa22b088c5565ee49d8281156078f SystemUUID:ec2fa22b-088c-5565-ee49-d8281156078f BootID:a02f36ca-004e-4b38-922e-57fa15c488d2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9f:68:53:b5:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9f:68:53:b5:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:27:2e:c9:94:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 23:08:07.144258 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.144254 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 23:08:07.144396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.144375 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 23:08:07.145667 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.145632 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 23:08:07.145886 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.145671 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 23:08:07.146042 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.146024 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:08:07.146703 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.146690 2571 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 23:08:07.146741 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.146707 2571 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 23:08:07.146741 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.146718 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:08:07.147334 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.147321 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 23:08:07.148932 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.148916 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 23:08:07.153678 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153657 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:08:07.153844 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153833 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 23:08:07.153884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153875 2571 kubelet.go:486] "Attempting to sync node with API server" Sep 09 23:08:07.153914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153899 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 23:08:07.153942 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153919 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 23:08:07.153942 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153933 2571 kubelet.go:398] "Adding apiserver pod source" Sep 09 23:08:07.154020 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.153945 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 23:08:07.154439 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.154419 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:08:07.154439 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.154445 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 23:08:07.156830 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.156802 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 23:08:07.158976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.158962 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 23:08:07.159040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.158992 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 23:08:07.159155 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159145 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 23:08:07.159186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159160 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 23:08:07.159186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159166 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 23:08:07.159186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159172 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 23:08:07.159186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159179 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 23:08:07.159186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159184 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 23:08:07.159574 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159564 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 23:08:07.159574 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159574 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 23:08:07.159630 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159582 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 23:08:07.159630 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159587 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 23:08:07.159630 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159597 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 23:08:07.159630 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.159604 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 23:08:07.160951 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.160938 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 23:08:07.160951 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.160952 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 23:08:07.165129 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.165112 2571 server.go:1268] "Started kubelet" Sep 09 23:08:07.165366 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.165262 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 23:08:07.165480 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.165395 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 23:08:07.165581 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.165483 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 23:08:07.166296 ip-10-0-140-142 systemd[1]: Started Kubernetes Kubelet. Sep 09 23:08:07.166685 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.166665 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 23:08:07.173662 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.173620 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 23:08:07.174043 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.174025 2571 server.go:310] "Adding debug handlers to kubelet server" Sep 09 23:08:07.176685 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.176658 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 23:08:07.176799 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.174845 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-142.ec2.internal.18d3c9bc3452b156 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-142.ec2.internal,UID:ip-10-0-140-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-142.ec2.internal,},FirstTimestamp:2026-09-09 23:08:07.165088086 +0000 UTC m=+0.337376162,LastTimestamp:2026-09-09 23:08:07.165088086 +0000 UTC m=+0.337376162,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-142.ec2.internal,}" Sep 09 23:08:07.176799 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.176772 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 23:08:07.176922 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.176841 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 23:08:07.181382 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.181354 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 23:08:07.181807 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.181780 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 23:08:07.182126 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.182105 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.182692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.182458 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 23:08:07.182692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.182478 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 23:08:07.182692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.182488 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 23:08:07.182692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.182584 2571 reconstruct.go:97] "Volume reconstruction finished" Sep 09 23:08:07.182692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.182591 2571 reconciler.go:29] "Reconciler: start to sync state" Sep 09 23:08:07.183143 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183125 2571 factory.go:55] Registering systemd factory Sep 09 23:08:07.183190 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183170 2571 factory.go:223] Registration of the systemd container factory successfully Sep 09 23:08:07.183480 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183466 2571 factory.go:153] Registering CRI-O factory Sep 09 23:08:07.183529 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183483 2571 factory.go:223] Registration of the crio container factory successfully Sep 09 23:08:07.183575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183540 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 23:08:07.183575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183560 2571 factory.go:103] Registering Raw factory Sep 09 23:08:07.183575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.183575 2571 manager.go:1196] Started watching for new ooms in manager Sep 09 23:08:07.184035 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.184023 2571 manager.go:349] Starting recovery of all containers Sep 09 23:08:07.184353 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.184223 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 23:08:07.184431 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.184355 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 23:08:07.192515 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.192293 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qpmrp" Sep 09 23:08:07.194519 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.194499 2571 manager.go:354] Recovery completed Sep 09 23:08:07.198722 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.198703 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.199350 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.199330 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qpmrp" Sep 09 23:08:07.200415 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.200395 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.200513 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.200465 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.200513 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.200480 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.201128 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201114 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 23:08:07.201128 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201122 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.201128 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201128 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 23:08:07.201276 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201147 2571 state_mem.go:36] "Initialized new in-memory state store" Sep 09 23:08:07.201792 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201775 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.201856 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201803 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.201856 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.201813 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.202951 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.202937 2571 policy_none.go:49] "None policy: Start" Sep 09 23:08:07.203014 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.202957 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 23:08:07.203014 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.202977 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 23:08:07.205298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.205223 2571 policy_none.go:47] "Start" Sep 09 23:08:07.249992 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.249961 2571 manager.go:337] "Starting Device Plugin manager" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.250013 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.250024 2571 server.go:87] "Starting device plugin registration server" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.250411 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.250423 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.250530 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.250627 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.250639 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.251159 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 23:08:07.262286 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.251204 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.317386 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.317339 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 23:08:07.318524 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.318500 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 23:08:07.318645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.318556 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 23:08:07.318645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.318582 2571 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 23:08:07.318645 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.318623 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 23:08:07.321483 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.321461 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:08:07.350889 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.350784 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.351864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.351846 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.351962 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.351883 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.351962 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.351899 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.351962 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.351929 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.361587 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.361558 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.361710 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.361595 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-142.ec2.internal\": node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.385157 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.385127 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.419255 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.419195 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal"] Sep 09 23:08:07.419418 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.419296 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.420810 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.420790 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.420914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.420819 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.420914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.420829 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.423177 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423163 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.423367 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423353 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.423408 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423384 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.423964 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423948 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.423964 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423961 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.424108 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423986 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.424108 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.423998 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.424108 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.424000 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.424108 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.424010 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.426230 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.426213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.426408 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.426251 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 23:08:07.426978 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.426962 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientMemory" Sep 09 23:08:07.427072 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.426990 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 23:08:07.427072 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.427004 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeHasSufficientPID" Sep 09 23:08:07.450464 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.450433 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-142.ec2.internal\" not found" node="ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.455000 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.454979 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-142.ec2.internal\" not found" node="ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.484165 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.484130 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa22e1d5883b0f6d0de711b180c14346-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal\" (UID: \"fa22e1d5883b0f6d0de711b180c14346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.484165 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.484163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa22e1d5883b0f6d0de711b180c14346-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal\" (UID: \"fa22e1d5883b0f6d0de711b180c14346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.484397 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.484187 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18d764b0ed3aa25d3bd2d54df9d49711-config\") pod \"kube-apiserver-proxy-ip-10-0-140-142.ec2.internal\" (UID: \"18d764b0ed3aa25d3bd2d54df9d49711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.486252 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.486217 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.584687 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.584635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa22e1d5883b0f6d0de711b180c14346-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal\" (UID: \"fa22e1d5883b0f6d0de711b180c14346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.584687 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.584685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa22e1d5883b0f6d0de711b180c14346-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal\" (UID: \"fa22e1d5883b0f6d0de711b180c14346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.584687 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.584703 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18d764b0ed3aa25d3bd2d54df9d49711-config\") pod \"kube-apiserver-proxy-ip-10-0-140-142.ec2.internal\" (UID: \"18d764b0ed3aa25d3bd2d54df9d49711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.584950 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.584750 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/18d764b0ed3aa25d3bd2d54df9d49711-config\") pod \"kube-apiserver-proxy-ip-10-0-140-142.ec2.internal\" (UID: \"18d764b0ed3aa25d3bd2d54df9d49711\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.584950 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.584754 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa22e1d5883b0f6d0de711b180c14346-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal\" (UID: \"fa22e1d5883b0f6d0de711b180c14346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.584950 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.584761 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa22e1d5883b0f6d0de711b180c14346-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal\" (UID: \"fa22e1d5883b0f6d0de711b180c14346\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.586758 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.586736 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.687606 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.687528 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.752674 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.752640 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.757704 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:07.757677 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" Sep 09 23:08:07.788635 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.788596 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.889143 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.889109 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:07.989800 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:07.989702 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:08.076133 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.076099 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:08:08.090824 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:08.090785 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:08.117940 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.112955 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 23:08:08.117940 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.113114 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:08:08.117940 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.113148 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 23:08:08.191425 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:08.191387 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:08.201768 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.201732 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 23:03:08 +0000 UTC" deadline="2028-06-13 19:47:41.100240054 +0000 UTC" Sep 09 23:08:08.201768 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.201764 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15428h39m32.898479591s" Sep 09 23:08:08.292380 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:08.292345 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:08.392966 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:08.392927 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:08.433344 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:08.433292 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa22e1d5883b0f6d0de711b180c14346.slice/crio-38106069fb274ce7b2cac4b88dc6bf1428bece463cef22f7f7b0aa6ce085d7ab WatchSource:0}: Error finding container 38106069fb274ce7b2cac4b88dc6bf1428bece463cef22f7f7b0aa6ce085d7ab: Status 404 returned error can't find the container with id 38106069fb274ce7b2cac4b88dc6bf1428bece463cef22f7f7b0aa6ce085d7ab Sep 09 23:08:08.433624 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:08.433602 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d764b0ed3aa25d3bd2d54df9d49711.slice/crio-88b0a278533f2bb5d797b2019270780317397bc49eabebf980609c1a30a4e855 WatchSource:0}: Error finding container 88b0a278533f2bb5d797b2019270780317397bc49eabebf980609c1a30a4e855: Status 404 returned error can't find the container with id 88b0a278533f2bb5d797b2019270780317397bc49eabebf980609c1a30a4e855 Sep 09 23:08:08.439593 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.439577 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:08:08.493170 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:08.493129 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-142.ec2.internal\" not found" Sep 09 23:08:08.554233 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.554155 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:08:08.563064 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.563032 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:08:08.582953 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.582921 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" Sep 09 23:08:08.598443 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.598418 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:08:08.599320 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.599305 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" Sep 09 23:08:08.608574 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:08.608552 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 23:08:09.154435 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.154391 2571 apiserver.go:52] "Watching apiserver" Sep 09 23:08:09.166400 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.166365 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 23:08:09.166832 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.166802 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qp8cl","openshift-cluster-node-tuning-operator/tuned-zvxfl","openshift-dns/node-resolver-tg9zt","openshift-image-registry/node-ca-v7v25","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal","openshift-multus/multus-additional-cni-plugins-sg94k","openshift-network-diagnostics/network-check-target-s2tnq","openshift-network-operator/iptables-alerter-7dcgs","kube-system/konnectivity-agent-sv6q5","kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg","openshift-multus/multus-q9z2x","openshift-multus/network-metrics-daemon-xbcm2","openshift-ovn-kubernetes/ovnkube-node-km998"] Sep 09 23:08:09.169372 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.169343 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:09.169490 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.169427 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:09.171747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.171718 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.174064 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.174036 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.175157 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.175137 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.175157 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.175150 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d96fs\"" Sep 09 23:08:09.175481 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.175464 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.175762 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.175743 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 23:08:09.176032 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.176011 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.177550 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.177528 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.177550 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.177547 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k6kfx\"" Sep 09 23:08:09.178621 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.178308 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.178621 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.178595 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.178976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.178957 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.179279 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.179150 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.179584 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.179539 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 23:08:09.179677 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.179584 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pz5xr\"" Sep 09 23:08:09.181647 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.181127 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.181647 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.181283 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.181647 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.181382 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47jvd\"" Sep 09 23:08:09.181855 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.181651 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 23:08:09.181855 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.181682 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 23:08:09.182145 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.182006 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.182273 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.182216 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 23:08:09.183645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.183012 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 23:08:09.183645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.183103 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:08:09.183645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.183119 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 23:08:09.183846 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.183742 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.185493 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.185471 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.185699 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.185683 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 23:08:09.185887 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.185870 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wcrbr\"" Sep 09 23:08:09.187208 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.186149 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.187208 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.186438 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 23:08:09.187208 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.186695 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 23:08:09.189154 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.188108 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lls7q\"" Sep 09 23:08:09.189904 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.189886 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 23:08:09.190054 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.190038 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:09.190323 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.190295 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:09.190405 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.190078 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 23:08:09.192495 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192401 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-hosts-file\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.192597 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-cnibin\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.192597 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.192702 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-socket-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.192702 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-tmp-dir\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.192702 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192670 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/381e539a-f46d-4ce6-b093-8192a74feb89-konnectivity-ca\") pod \"konnectivity-agent-sv6q5\" (UID: \"381e539a-f46d-4ce6-b093-8192a74feb89\") " pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.192853 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovnkube-config\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.192853 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.192853 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192819 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-env-overrides\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.192989 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-device-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.192989 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192886 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.192989 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192914 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-sys-fs\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.192989 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.192950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2l75\" (UniqueName: \"kubernetes.io/projected/10454f55-e51e-421f-b664-79d547176e47-kube-api-access-h2l75\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.193182 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193010 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86cd631b-4036-4398-b0a6-63d6c08bb6b8-serviceca\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.193182 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193043 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-systemd\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193182 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9p6n\" (UniqueName: \"kubernetes.io/projected/855335ea-12b5-4cfd-b507-eafb20d3e93e-kube-api-access-k9p6n\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193182 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193135 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:09.193385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193203 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.193385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.193385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-node-log\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovnkube-script-lib\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193413 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpx27\" (UniqueName: \"kubernetes.io/projected/ecd37d40-2a05-4131-b63b-7dd542794f8a-kube-api-access-wpx27\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.193575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193450 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/381e539a-f46d-4ce6-b093-8192a74feb89-agent-certs\") pod \"konnectivity-agent-sv6q5\" (UID: \"381e539a-f46d-4ce6-b093-8192a74feb89\") " pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.193575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193484 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86cd631b-4036-4398-b0a6-63d6c08bb6b8-host\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.193575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-slash\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-etc-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193586 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193608 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-log-socket\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctx6\" (UniqueName: \"kubernetes.io/projected/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-kube-api-access-lctx6\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193693 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-system-cni-dir\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.193766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193765 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5sx8\" (UniqueName: \"kubernetes.io/projected/86cd631b-4036-4398-b0a6-63d6c08bb6b8-kube-api-access-n5sx8\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-run-ovn-kubernetes\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193830 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193860 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-registration-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193915 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-run-netns\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.193979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-var-lib-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194054 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-ovn\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194069 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194056 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.194449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-cni-bin\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194125 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-cni-netd\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovn-node-metrics-cert\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194259 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-kubelet\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.194634 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.194489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-systemd-units\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.195881 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.195604 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.197316 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.197293 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.197540 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.197525 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.197881 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.197861 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pnwbt\"" Sep 09 23:08:09.198643 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.198620 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 23:08:09.198743 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.198723 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 23:08:09.198851 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.198834 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 23:08:09.198950 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.198935 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8sz4w\"" Sep 09 23:08:09.199198 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.199177 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.199364 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.199340 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:09.201562 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.201540 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.204203 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.204183 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 23:08:09.204451 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.204433 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hp2hq\"" Sep 09 23:08:09.220203 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.220174 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ctdfr" Sep 09 23:08:09.226113 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.226092 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ctdfr" Sep 09 23:08:09.284010 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.283876 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 23:08:09.295596 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/381e539a-f46d-4ce6-b093-8192a74feb89-agent-certs\") pod \"konnectivity-agent-sv6q5\" (UID: \"381e539a-f46d-4ce6-b093-8192a74feb89\") " pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.295596 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86cd631b-4036-4398-b0a6-63d6c08bb6b8-host\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295625 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-etc-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-cni-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295675 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-cni-multus\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctx6\" (UniqueName: \"kubernetes.io/projected/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-kube-api-access-lctx6\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295709 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/86cd631b-4036-4398-b0a6-63d6c08bb6b8-host\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295762 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-etc-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295798 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5sx8\" (UniqueName: \"kubernetes.io/projected/86cd631b-4036-4398-b0a6-63d6c08bb6b8-kube-api-access-n5sx8\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.295825 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-host\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81ca7c58-f6cf-45d2-892c-fbb04006ca41-cni-binary-copy\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295852 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-netns\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295869 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2twfj\" (UniqueName: \"kubernetes.io/projected/81ca7c58-f6cf-45d2-892c-fbb04006ca41-kube-api-access-2twfj\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.295921 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d428ece3-294f-40dc-aea8-1fd7db6f9531-host-slash\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-var-lib-kubelet\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-run-netns\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296114 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-var-lib-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296142 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-run-netns\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296186 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-cni-bin\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296202 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-var-lib-openvswitch\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296222 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-cni-netd\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovn-node-metrics-cert\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-cni-bin\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.296298 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296292 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-daemon-config\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296308 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-cni-netd\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296316 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-kubelet\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296340 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-cnibin\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296368 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-kubelet\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296361 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-k8s-cni-cncf-io\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296409 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-etc-kubernetes\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296424 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296432 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-socket-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/381e539a-f46d-4ce6-b093-8192a74feb89-konnectivity-ca\") pod \"konnectivity-agent-sv6q5\" (UID: \"381e539a-f46d-4ce6-b093-8192a74feb89\") " pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296530 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-run\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296542 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-socket-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-system-cni-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296582 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-kubelet\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296676 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.296999 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/381e539a-f46d-4ce6-b093-8192a74feb89-konnectivity-ca\") pod \"konnectivity-agent-sv6q5\" (UID: \"381e539a-f46d-4ce6-b093-8192a74feb89\") " pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tqd8\" (UniqueName: \"kubernetes.io/projected/d428ece3-294f-40dc-aea8-1fd7db6f9531-kube-api-access-6tqd8\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.297067 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297033 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-dbus\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-env-overrides\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297111 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-tuned\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297176 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-tmp\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297200 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297229 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2l75\" (UniqueName: \"kubernetes.io/projected/10454f55-e51e-421f-b664-79d547176e47-kube-api-access-h2l75\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297293 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-systemd\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297316 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9p6n\" (UniqueName: \"kubernetes.io/projected/855335ea-12b5-4cfd-b507-eafb20d3e93e-kube-api-access-k9p6n\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297319 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-etc-selinux\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-lib-modules\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297367 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-systemd\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297406 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-node-log\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.297884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovnkube-script-lib\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297483 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-hostroot\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297505 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-env-overrides\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-multus-certs\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpx27\" (UniqueName: \"kubernetes.io/projected/ecd37d40-2a05-4131-b63b-7dd542794f8a-kube-api-access-wpx27\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297573 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-slash\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-log-socket\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297619 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-kubelet-config\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-system-cni-dir\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-run-ovn-kubernetes\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297707 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-conf-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297755 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysctl-d\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297777 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysctl-conf\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-sys\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-registration-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297840 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.298679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297849 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-ovn\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297872 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-modprobe-d\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297892 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-node-log\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-socket-dir-parent\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpnq\" (UniqueName: \"kubernetes.io/projected/8647901c-c88a-4922-a37b-ecff32c52f6f-kube-api-access-7zpnq\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-systemd\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.297998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-systemd-units\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-hosts-file\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298049 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-cnibin\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298095 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d428ece3-294f-40dc-aea8-1fd7db6f9531-iptables-alerter-script\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298117 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-slash\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298171 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-cnibin\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298181 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-run-ovn-kubernetes\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298226 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-hosts-file\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298264 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovnkube-script-lib\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298284 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.299539 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298339 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-registration-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298342 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-run-ovn\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-tmp-dir\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovnkube-config\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298472 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-log-socket\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298515 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ecd37d40-2a05-4131-b63b-7dd542794f8a-system-cni-dir\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5ts\" (UniqueName: \"kubernetes.io/projected/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-kube-api-access-8r5ts\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298584 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-cni-bin\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298608 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298622 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298642 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-tmp-dir\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-device-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298681 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/855335ea-12b5-4cfd-b507-eafb20d3e93e-systemd-units\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-sys-fs\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298750 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86cd631b-4036-4398-b0a6-63d6c08bb6b8-serviceca\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298789 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-sys-fs\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.300351 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298817 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysconfig\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.301167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298856 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-kubernetes\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.301167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.298753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10454f55-e51e-421f-b664-79d547176e47-device-dir\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.301167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.299107 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovnkube-config\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.301167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.299232 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/86cd631b-4036-4398-b0a6-63d6c08bb6b8-serviceca\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.301167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.299792 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ecd37d40-2a05-4131-b63b-7dd542794f8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.301428 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.301218 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/381e539a-f46d-4ce6-b093-8192a74feb89-agent-certs\") pod \"konnectivity-agent-sv6q5\" (UID: \"381e539a-f46d-4ce6-b093-8192a74feb89\") " pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.301428 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.301323 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/855335ea-12b5-4cfd-b507-eafb20d3e93e-ovn-node-metrics-cert\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.305831 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.305182 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:09.305831 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.305216 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:09.305831 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.305253 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:09.305831 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.305376 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:08:09.805331836 +0000 UTC m=+2.977619924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:09.309998 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.309627 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2l75\" (UniqueName: \"kubernetes.io/projected/10454f55-e51e-421f-b664-79d547176e47-kube-api-access-h2l75\") pod \"aws-ebs-csi-driver-node-ptrlg\" (UID: \"10454f55-e51e-421f-b664-79d547176e47\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.312742 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.312450 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpx27\" (UniqueName: \"kubernetes.io/projected/ecd37d40-2a05-4131-b63b-7dd542794f8a-kube-api-access-wpx27\") pod \"multus-additional-cni-plugins-sg94k\" (UID: \"ecd37d40-2a05-4131-b63b-7dd542794f8a\") " pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.314965 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.314757 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5sx8\" (UniqueName: \"kubernetes.io/projected/86cd631b-4036-4398-b0a6-63d6c08bb6b8-kube-api-access-n5sx8\") pod \"node-ca-v7v25\" (UID: \"86cd631b-4036-4398-b0a6-63d6c08bb6b8\") " pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.316197 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.316170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctx6\" (UniqueName: \"kubernetes.io/projected/2ae6772e-e29f-49ed-b872-0c2f6b30ac90-kube-api-access-lctx6\") pod \"node-resolver-tg9zt\" (UID: \"2ae6772e-e29f-49ed-b872-0c2f6b30ac90\") " pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.316941 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.316888 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9p6n\" (UniqueName: \"kubernetes.io/projected/855335ea-12b5-4cfd-b507-eafb20d3e93e-kube-api-access-k9p6n\") pod \"ovnkube-node-km998\" (UID: \"855335ea-12b5-4cfd-b507-eafb20d3e93e\") " pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.323906 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.323846 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" event={"ID":"fa22e1d5883b0f6d0de711b180c14346","Type":"ContainerStarted","Data":"38106069fb274ce7b2cac4b88dc6bf1428bece463cef22f7f7b0aa6ce085d7ab"} Sep 09 23:08:09.325693 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.325661 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" event={"ID":"18d764b0ed3aa25d3bd2d54df9d49711","Type":"ContainerStarted","Data":"88b0a278533f2bb5d797b2019270780317397bc49eabebf980609c1a30a4e855"} Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d428ece3-294f-40dc-aea8-1fd7db6f9531-iptables-alerter-script\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5ts\" (UniqueName: \"kubernetes.io/projected/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-kube-api-access-8r5ts\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-cni-bin\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400834 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400865 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysconfig\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-kubernetes\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-cni-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400939 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-cni-multus\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.400964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-host\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81ca7c58-f6cf-45d2-892c-fbb04006ca41-cni-binary-copy\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401025 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-netns\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401049 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2twfj\" (UniqueName: \"kubernetes.io/projected/81ca7c58-f6cf-45d2-892c-fbb04006ca41-kube-api-access-2twfj\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401073 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d428ece3-294f-40dc-aea8-1fd7db6f9531-host-slash\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-var-lib-kubelet\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-daemon-config\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-cnibin\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-k8s-cni-cncf-io\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.403709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-etc-kubernetes\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401218 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-run\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-system-cni-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-kubelet\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tqd8\" (UniqueName: \"kubernetes.io/projected/d428ece3-294f-40dc-aea8-1fd7db6f9531-kube-api-access-6tqd8\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-dbus\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401352 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-tuned\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-tmp\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401428 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-lib-modules\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-hostroot\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-multus-certs\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-kubelet-config\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-conf-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401572 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysctl-d\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysctl-conf\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401616 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-sys\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401640 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-modprobe-d\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.404883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401677 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-socket-dir-parent\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpnq\" (UniqueName: \"kubernetes.io/projected/8647901c-c88a-4922-a37b-ecff32c52f6f-kube-api-access-7zpnq\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401729 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-systemd\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.401871 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-systemd\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402286 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-netns\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402515 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d428ece3-294f-40dc-aea8-1fd7db6f9531-iptables-alerter-script\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402584 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d428ece3-294f-40dc-aea8-1fd7db6f9531-host-slash\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402643 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-var-lib-kubelet\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402664 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-system-cni-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402726 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-kubelet\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402752 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81ca7c58-f6cf-45d2-892c-fbb04006ca41-cni-binary-copy\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402777 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-k8s-cni-cncf-io\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402812 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-run\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402839 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-cnibin\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402884 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-etc-kubernetes\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402931 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-kubelet-config\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402983 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-dbus\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.402973 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-conf-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.405766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403084 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-daemon-config\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403088 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-cni-bin\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403108 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysctl-d\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.403155 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403173 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-cni-dir\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403226 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysconfig\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403227 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-sysctl-conf\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.403231 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:08:09.903212173 +0000 UTC m=+3.075500254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403290 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-kubernetes\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403306 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-host\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403315 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-sys\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403350 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-var-lib-cni-multus\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403380 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-lib-modules\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403447 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-modprobe-d\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403495 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-multus-socket-dir-parent\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.403533 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:09.403574 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:09.903560948 +0000 UTC m=+3.075849027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:09.406605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403618 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-hostroot\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.407432 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.403664 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81ca7c58-f6cf-45d2-892c-fbb04006ca41-host-run-multus-certs\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.407432 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.406969 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-tmp\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.407432 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.407013 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-etc-tuned\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.414608 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.414578 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2twfj\" (UniqueName: \"kubernetes.io/projected/81ca7c58-f6cf-45d2-892c-fbb04006ca41-kube-api-access-2twfj\") pod \"multus-q9z2x\" (UID: \"81ca7c58-f6cf-45d2-892c-fbb04006ca41\") " pod="openshift-multus/multus-q9z2x" Sep 09 23:08:09.414783 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.414754 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tqd8\" (UniqueName: \"kubernetes.io/projected/d428ece3-294f-40dc-aea8-1fd7db6f9531-kube-api-access-6tqd8\") pod \"iptables-alerter-7dcgs\" (UID: \"d428ece3-294f-40dc-aea8-1fd7db6f9531\") " pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:09.415130 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.415094 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5ts\" (UniqueName: \"kubernetes.io/projected/0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb-kube-api-access-8r5ts\") pod \"tuned-zvxfl\" (UID: \"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb\") " pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:09.415812 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.415785 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpnq\" (UniqueName: \"kubernetes.io/projected/8647901c-c88a-4922-a37b-ecff32c52f6f-kube-api-access-7zpnq\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:09.488885 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.488804 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" Sep 09 23:08:09.495939 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.495911 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tg9zt" Sep 09 23:08:09.508071 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.508037 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v7v25" Sep 09 23:08:09.513662 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.513632 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:09.521438 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.521407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sg94k" Sep 09 23:08:09.528423 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.528369 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:09.535361 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:09.535334 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" Sep 09 23:08:10.768049 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:10.768019 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7dcgs" Sep 09 23:08:10.773971 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:10.773939 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9z2x" Sep 09 23:08:11.030449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:11.030372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:11.030583 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.030504 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:11.030583 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.030523 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:11.030583 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.030533 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:11.030695 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.030589 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:08:12.030573544 +0000 UTC m=+3.977985244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:11.130967 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:11.130924 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:11.131157 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:11.131013 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:11.131157 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.131091 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:11.131255 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.131156 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:11.131255 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.131185 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:12.131169694 +0000 UTC m=+4.078581395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:11.131255 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:11.131211 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:08:12.13119451 +0000 UTC m=+4.078606214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:11.454334 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:11.454249 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:03:10 +0000 UTC" deadline="2028-06-24 14:25:03.044072874 +0000 UTC" Sep 09 23:08:11.454334 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:11.454284 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15687h16m51.589792986s" Sep 09 23:08:11.681892 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.681859 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da26e8d_a4da_4ffe_87a3_cddbf1d3d6fb.slice/crio-20c7ce1ef5b3a6db28770a3a7193bfd4cb571a990752b1749925951e6efca8a1 WatchSource:0}: Error finding container 20c7ce1ef5b3a6db28770a3a7193bfd4cb571a990752b1749925951e6efca8a1: Status 404 returned error can't find the container with id 20c7ce1ef5b3a6db28770a3a7193bfd4cb571a990752b1749925951e6efca8a1 Sep 09 23:08:11.687251 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.687219 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cd631b_4036_4398_b0a6_63d6c08bb6b8.slice/crio-fcb9448adeba224a2bd8ba8bc0c42603eb4e0f7c5ea7d4bfeea10e7026b8c1c2 WatchSource:0}: Error finding container fcb9448adeba224a2bd8ba8bc0c42603eb4e0f7c5ea7d4bfeea10e7026b8c1c2: Status 404 returned error can't find the container with id fcb9448adeba224a2bd8ba8bc0c42603eb4e0f7c5ea7d4bfeea10e7026b8c1c2 Sep 09 23:08:11.688318 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.688291 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd428ece3_294f_40dc_aea8_1fd7db6f9531.slice/crio-d6175abbc8ba79cde3935c7e3fb88e7ca2ecc8f2c3d9efed1fd3cd2057d88120 WatchSource:0}: Error finding container d6175abbc8ba79cde3935c7e3fb88e7ca2ecc8f2c3d9efed1fd3cd2057d88120: Status 404 returned error can't find the container with id d6175abbc8ba79cde3935c7e3fb88e7ca2ecc8f2c3d9efed1fd3cd2057d88120 Sep 09 23:08:11.689745 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.689719 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855335ea_12b5_4cfd_b507_eafb20d3e93e.slice/crio-ee60b06629b5fea91381299f12e9758f84e8b2a4d66a4872c82b20c6b0c2030b WatchSource:0}: Error finding container ee60b06629b5fea91381299f12e9758f84e8b2a4d66a4872c82b20c6b0c2030b: Status 404 returned error can't find the container with id ee60b06629b5fea91381299f12e9758f84e8b2a4d66a4872c82b20c6b0c2030b Sep 09 23:08:11.690330 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.690305 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae6772e_e29f_49ed_b872_0c2f6b30ac90.slice/crio-2f72934177afc946fb67c6772933954d281a8861c668d12dc4f3034de2471e71 WatchSource:0}: Error finding container 2f72934177afc946fb67c6772933954d281a8861c668d12dc4f3034de2471e71: Status 404 returned error can't find the container with id 2f72934177afc946fb67c6772933954d281a8861c668d12dc4f3034de2471e71 Sep 09 23:08:11.693712 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.692452 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd37d40_2a05_4131_b63b_7dd542794f8a.slice/crio-c04f38357e44331e0b895480210ff891df507135fb3b796733e5fc41d8ee1280 WatchSource:0}: Error finding container c04f38357e44331e0b895480210ff891df507135fb3b796733e5fc41d8ee1280: Status 404 returned error can't find the container with id c04f38357e44331e0b895480210ff891df507135fb3b796733e5fc41d8ee1280 Sep 09 23:08:11.693712 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.692937 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ca7c58_f6cf_45d2_892c_fbb04006ca41.slice/crio-3ce2d7fba392912e0bde022cff106487a08c3963e621aa5f6499c0e2a20439c8 WatchSource:0}: Error finding container 3ce2d7fba392912e0bde022cff106487a08c3963e621aa5f6499c0e2a20439c8: Status 404 returned error can't find the container with id 3ce2d7fba392912e0bde022cff106487a08c3963e621aa5f6499c0e2a20439c8 Sep 09 23:08:11.696085 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.695717 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10454f55_e51e_421f_b664_79d547176e47.slice/crio-1d785b24a7dee8a2913439f7c68d5e0c40db05b3eb7480a5cb09921bf1294eb4 WatchSource:0}: Error finding container 1d785b24a7dee8a2913439f7c68d5e0c40db05b3eb7480a5cb09921bf1294eb4: Status 404 returned error can't find the container with id 1d785b24a7dee8a2913439f7c68d5e0c40db05b3eb7480a5cb09921bf1294eb4 Sep 09 23:08:11.697911 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:11.697545 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381e539a_f46d_4ce6_b093_8192a74feb89.slice/crio-c264857e26c52359d388e15f0d6fc96fa7915e4fa1c4e46ed7a4755ada5ed310 WatchSource:0}: Error finding container c264857e26c52359d388e15f0d6fc96fa7915e4fa1c4e46ed7a4755ada5ed310: Status 404 returned error can't find the container with id c264857e26c52359d388e15f0d6fc96fa7915e4fa1c4e46ed7a4755ada5ed310 Sep 09 23:08:12.036905 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.036822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:12.037034 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.036956 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:12.037034 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.036973 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:12.037034 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.036981 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:12.037034 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.037033 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:08:14.037019129 +0000 UTC m=+5.984430849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:12.137588 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.137548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:12.137777 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.137603 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:12.137777 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.137705 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:12.137777 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.137712 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:12.137777 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.137758 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:14.137745692 +0000 UTC m=+6.085157394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:12.137954 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.137782 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:08:14.137764003 +0000 UTC m=+6.085175704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:12.407871 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.407766 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 23:08:12.455381 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.455305 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 23:03:10 +0000 UTC" deadline="2028-03-18 13:12:22.97320161 +0000 UTC" Sep 09 23:08:12.455381 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.455366 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13334h4m10.517840239s" Sep 09 23:08:12.545150 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.543704 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:12.545150 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.543841 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:12.545150 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.544253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:12.545150 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.544351 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:12.545150 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.544414 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:12.545150 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:12.544490 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:12.571519 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.571478 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" event={"ID":"18d764b0ed3aa25d3bd2d54df9d49711","Type":"ContainerStarted","Data":"5a30235a03229ff9187eef1efcfaef673f79721e7b6463a2097ec53e221e2a69"} Sep 09 23:08:12.580952 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.580915 2571 generic.go:356] "Generic (PLEG): container finished" podID="fa22e1d5883b0f6d0de711b180c14346" containerID="23b73c0a616b24831163565bf0a2564fe8a9c318271d030abbe8e5acf2ae2a2f" exitCode=0 Sep 09 23:08:12.581134 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.581031 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" event={"ID":"fa22e1d5883b0f6d0de711b180c14346","Type":"ContainerDied","Data":"23b73c0a616b24831163565bf0a2564fe8a9c318271d030abbe8e5acf2ae2a2f"} Sep 09 23:08:12.589390 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.589350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" event={"ID":"10454f55-e51e-421f-b664-79d547176e47","Type":"ContainerStarted","Data":"1d785b24a7dee8a2913439f7c68d5e0c40db05b3eb7480a5cb09921bf1294eb4"} Sep 09 23:08:12.596978 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.596854 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerStarted","Data":"c04f38357e44331e0b895480210ff891df507135fb3b796733e5fc41d8ee1280"} Sep 09 23:08:12.605534 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.605466 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-142.ec2.internal" podStartSLOduration=3.605443698 podStartE2EDuration="3.605443698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:12.588425439 +0000 UTC m=+4.535837162" watchObservedRunningTime="2026-09-09 23:08:12.605443698 +0000 UTC m=+4.552855405" Sep 09 23:08:12.619490 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.619446 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tg9zt" event={"ID":"2ae6772e-e29f-49ed-b872-0c2f6b30ac90","Type":"ContainerStarted","Data":"2f72934177afc946fb67c6772933954d281a8861c668d12dc4f3034de2471e71"} Sep 09 23:08:12.623223 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.623184 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"ee60b06629b5fea91381299f12e9758f84e8b2a4d66a4872c82b20c6b0c2030b"} Sep 09 23:08:12.626136 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.626083 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7dcgs" event={"ID":"d428ece3-294f-40dc-aea8-1fd7db6f9531","Type":"ContainerStarted","Data":"d6175abbc8ba79cde3935c7e3fb88e7ca2ecc8f2c3d9efed1fd3cd2057d88120"} Sep 09 23:08:12.632764 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.632660 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sv6q5" event={"ID":"381e539a-f46d-4ce6-b093-8192a74feb89","Type":"ContainerStarted","Data":"c264857e26c52359d388e15f0d6fc96fa7915e4fa1c4e46ed7a4755ada5ed310"} Sep 09 23:08:12.640251 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.640148 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9z2x" event={"ID":"81ca7c58-f6cf-45d2-892c-fbb04006ca41","Type":"ContainerStarted","Data":"3ce2d7fba392912e0bde022cff106487a08c3963e621aa5f6499c0e2a20439c8"} Sep 09 23:08:12.646001 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.645963 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v7v25" event={"ID":"86cd631b-4036-4398-b0a6-63d6c08bb6b8","Type":"ContainerStarted","Data":"fcb9448adeba224a2bd8ba8bc0c42603eb4e0f7c5ea7d4bfeea10e7026b8c1c2"} Sep 09 23:08:12.665976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:12.665803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" event={"ID":"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb","Type":"ContainerStarted","Data":"20c7ce1ef5b3a6db28770a3a7193bfd4cb571a990752b1749925951e6efca8a1"} Sep 09 23:08:13.679071 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:13.678987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" event={"ID":"fa22e1d5883b0f6d0de711b180c14346","Type":"ContainerStarted","Data":"794fdb080f1f30672fa5f3d740ada319f38b96ea1c4adb5dfc89491a4f7f9df7"} Sep 09 23:08:13.696062 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:13.695990 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-142.ec2.internal" podStartSLOduration=4.695967961 podStartE2EDuration="4.695967961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:13.695729162 +0000 UTC m=+5.643140886" watchObservedRunningTime="2026-09-09 23:08:13.695967961 +0000 UTC m=+5.643379685" Sep 09 23:08:14.053140 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:14.053034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:14.053320 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.053233 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:14.053320 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.053257 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:14.053320 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.053269 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:14.053481 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.053326 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:08:18.053307775 +0000 UTC m=+10.000719492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:14.153607 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:14.153565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:14.153779 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:14.153652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:14.153856 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.153778 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:14.153856 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.153838 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:08:18.153820429 +0000 UTC m=+10.101232137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:14.153966 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.153898 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:14.153966 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.153932 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:18.153920172 +0000 UTC m=+10.101331878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:14.556623 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:14.556584 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:14.556623 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:14.556603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:14.556864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:14.556603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:14.556864 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.556725 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:14.556966 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.556928 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:14.557055 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:14.557030 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:16.549037 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:16.549005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:16.549535 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:16.549150 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:16.549595 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:16.549561 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:16.549686 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:16.549666 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:16.549737 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:16.549726 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:16.549816 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:16.549798 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:18.087091 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:18.087048 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:18.087512 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.087269 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:18.087512 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.087289 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:18.087512 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.087302 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:18.087512 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.087367 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:08:26.087341594 +0000 UTC m=+18.034753298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:18.187817 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:18.187778 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:18.187989 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:18.187852 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:18.187989 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.187978 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:18.188199 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.188045 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:26.188026409 +0000 UTC m=+18.135438128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:18.188552 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.188352 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:18.188552 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.188403 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:08:26.188388918 +0000 UTC m=+18.135800634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:18.545166 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:18.544901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:18.545166 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.545022 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:18.545166 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:18.545069 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:18.545377 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:18.545181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:18.545377 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.545241 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:18.545377 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:18.545269 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:20.548598 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:20.548568 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:20.549032 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:20.548566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:20.549032 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:20.548683 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:20.549032 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:20.548719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:20.549032 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:20.548830 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:20.549032 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:20.548913 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:22.547412 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:22.547326 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:22.547864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:22.547450 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:22.547864 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:22.547460 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:22.547864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:22.547486 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:22.547864 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:22.547577 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:22.547864 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:22.547660 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:24.544127 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:24.543752 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:24.544127 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:24.543764 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:24.544127 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:24.543900 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:24.544127 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:24.543764 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:24.544127 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:24.543947 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:24.544127 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:24.544040 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:26.153510 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:26.153286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:26.154021 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.153471 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:26.154021 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.153567 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:26.154021 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.153583 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:26.154021 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.153640 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:08:42.153624954 +0000 UTC m=+34.101036655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:26.254392 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:26.254356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:26.254392 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:26.254413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:26.254649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.254523 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:26.254649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.254525 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:26.254649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.254592 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:42.254572278 +0000 UTC m=+34.201984204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:26.254649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.254612 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:08:42.254604535 +0000 UTC m=+34.202016236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:26.544755 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:26.544676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:26.544910 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:26.544676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:26.544910 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.544792 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:26.544910 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:26.544676 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:26.545027 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.544962 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:26.545027 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:26.544896 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:28.545740 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:28.545550 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:28.545740 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:28.545678 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:28.546282 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:28.545732 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:28.546282 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:28.545820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:28.546282 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:28.545935 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:28.546282 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:28.546054 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:29.716220 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.715941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" event={"ID":"10454f55-e51e-421f-b664-79d547176e47","Type":"ContainerStarted","Data":"99a0a2ae1a32273dd9a37322b55dddbd46182a2ccf98f01b1ebdaaa9dd0a2991"} Sep 09 23:08:29.717404 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.717378 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecd37d40-2a05-4131-b63b-7dd542794f8a" containerID="3578edb91ab0e9f2c12aee81d3da99ebe42e3e4fe9038bdd9d812a1961989bf4" exitCode=0 Sep 09 23:08:29.717486 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.717465 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerDied","Data":"3578edb91ab0e9f2c12aee81d3da99ebe42e3e4fe9038bdd9d812a1961989bf4"} Sep 09 23:08:29.718927 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.718851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tg9zt" event={"ID":"2ae6772e-e29f-49ed-b872-0c2f6b30ac90","Type":"ContainerStarted","Data":"b565c14f3a4164f00ed577a53196529ca308acf04364c5d6129f81f4ae8f222e"} Sep 09 23:08:29.721633 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.721610 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"34e471a1c05ab742664fea76d4f01804a230419ac167ff233cd11f2b093fd65b"} Sep 09 23:08:29.721741 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.721681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"3f257965cd2f3d569010345a5f72cb8917255cae10c424332b75248c04a0fcfc"} Sep 09 23:08:29.721741 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.721716 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"9181a1b3982b82f9d8c165c45ca690e3987ea6458c422a1b47a07bdc8839b911"} Sep 09 23:08:29.721824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.721755 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"681b8edeb37fd4ef1438534fdff8707b97a90022c7939e69814442c1f46796d4"} Sep 09 23:08:29.721824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.721783 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"6d5011cddf4ff6f745673e00de13a5177df18bcc07ce22ed0d8c7ca03f9cce2c"} Sep 09 23:08:29.723099 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.723076 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sv6q5" event={"ID":"381e539a-f46d-4ce6-b093-8192a74feb89","Type":"ContainerStarted","Data":"6998e5db62afd8ee3fc4cb660c2df03ee9f841149ce84356dba5919a957c19a1"} Sep 09 23:08:29.724625 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.724607 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9z2x" event={"ID":"81ca7c58-f6cf-45d2-892c-fbb04006ca41","Type":"ContainerStarted","Data":"45a68e2d9ece45df4118364c90250811826c1e64e608ea9728d09ff3a657ceca"} Sep 09 23:08:29.725983 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.725964 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v7v25" event={"ID":"86cd631b-4036-4398-b0a6-63d6c08bb6b8","Type":"ContainerStarted","Data":"434f078942a5a0ec7eb4883cc1fc3aab426723dc3bc73f7fd3998340811ad6ee"} Sep 09 23:08:29.727291 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.727273 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" event={"ID":"0da26e8d-a4da-4ffe-87a3-cddbf1d3d6fb","Type":"ContainerStarted","Data":"29054bda4a1671140c2827aa93c2bf3090812df9192a6b5d3c5b5581e1166961"} Sep 09 23:08:29.756552 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.756505 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v7v25" podStartSLOduration=4.23158927 podStartE2EDuration="21.756487343s" totalImagesPullingTime="17.524898073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:29.755839446 +0000 UTC m=+21.703251168" watchObservedRunningTime="2026-09-09 23:08:29.756487343 +0000 UTC m=+21.703899093" Sep 09 23:08:29.772999 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.772943 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tg9zt" podStartSLOduration=8.044432221 podStartE2EDuration="21.772927312s" totalImagesPullingTime="13.728495091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:29.772557312 +0000 UTC m=+21.719969035" watchObservedRunningTime="2026-09-09 23:08:29.772927312 +0000 UTC m=+21.720339034" Sep 09 23:08:29.792505 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.792453 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sv6q5" podStartSLOduration=8.075685012 podStartE2EDuration="21.792430626s" totalImagesPullingTime="13.716745614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:29.79218398 +0000 UTC m=+21.739595706" watchObservedRunningTime="2026-09-09 23:08:29.792430626 +0000 UTC m=+21.739842350" Sep 09 23:08:29.810868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.810819 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zvxfl" podStartSLOduration=4.279369382 podStartE2EDuration="21.810802967s" totalImagesPullingTime="17.531433585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:29.810539713 +0000 UTC m=+21.757951437" watchObservedRunningTime="2026-09-09 23:08:29.810802967 +0000 UTC m=+21.758214690" Sep 09 23:08:29.829333 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:29.829282 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q9z2x" podStartSLOduration=4.251228937 podStartE2EDuration="21.829266297s" totalImagesPullingTime="17.57803736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:29.828785595 +0000 UTC m=+21.776197329" watchObservedRunningTime="2026-09-09 23:08:29.829266297 +0000 UTC m=+21.776678019" Sep 09 23:08:30.544372 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.544271 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:30.544541 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:30.544416 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:30.544541 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.544271 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:30.544541 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:30.544531 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:30.544660 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.544607 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:30.544698 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:30.544676 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:30.735548 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.735373 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"d67d2cf563fcf9fafa76a70b0c4822fab9dfbd8ae6ec7dc937f89cee786d92a3"} Sep 09 23:08:30.736983 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.736951 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7dcgs" event={"ID":"d428ece3-294f-40dc-aea8-1fd7db6f9531","Type":"ContainerStarted","Data":"33be6f38b5ff94fbb3633c27b9a41a5ffe83e4c6183d42bd14a253657dc15d9e"} Sep 09 23:08:30.754188 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.754007 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7dcgs" podStartSLOduration=5.229527762 podStartE2EDuration="22.753986613s" totalImagesPullingTime="17.524458851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:30.753150029 +0000 UTC m=+22.700561753" watchObservedRunningTime="2026-09-09 23:08:30.753986613 +0000 UTC m=+22.701398337" Sep 09 23:08:30.854619 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:30.854581 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 23:08:31.487682 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.487518 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T23:08:30.854599265Z","UUID":"1363ac83-5682-4837-a154-78171ffccfbb","Handler":null,"Name":"","Endpoint":""} Sep 09 23:08:31.490585 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.490558 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 23:08:31.490737 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.490597 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 23:08:31.742562 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.742469 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" event={"ID":"10454f55-e51e-421f-b664-79d547176e47","Type":"ContainerStarted","Data":"80869e12a4f1b5b878ecacb47deade4a0e7dbacb27440848138c20d7563c4cc3"} Sep 09 23:08:31.884919 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.884891 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:31.885536 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.885518 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:31.890709 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.890654 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:31.891391 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:31.891372 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sv6q5" Sep 09 23:08:32.544385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:32.544277 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:32.544385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:32.544318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:32.544385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:32.544276 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:32.544649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:32.544397 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:32.544649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:32.544498 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:32.544649 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:32.544599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:32.747654 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:32.747614 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" event={"ID":"10454f55-e51e-421f-b664-79d547176e47","Type":"ContainerStarted","Data":"42f6e68e1095bfe9ac1e4a2e41db0ce3fd52926e1b814365d6a66be1ec41deaf"} Sep 09 23:08:32.754129 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:32.754076 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"60b4e7f21af5130c4ad1cb54e0e31b685fbd5cdf794461c84268ddbc06705174"} Sep 09 23:08:32.766645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:32.766584 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ptrlg" podStartSLOduration=4.748855333 podStartE2EDuration="24.766564037s" totalImagesPullingTime="20.017708704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:32.765983522 +0000 UTC m=+24.713395249" watchObservedRunningTime="2026-09-09 23:08:32.766564037 +0000 UTC m=+24.713975760" Sep 09 23:08:34.543926 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.543735 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:34.543926 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.543735 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:34.543926 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:34.543848 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:34.543926 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.543860 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:34.545284 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:34.543973 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:34.545284 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:34.544076 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:34.760718 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.760450 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerStarted","Data":"49c5a585f2a024d4513518dd5ab7f2f0f210be6995f6e380d31eb5cdc927958c"} Sep 09 23:08:34.763948 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.763922 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-km998" event={"ID":"855335ea-12b5-4cfd-b507-eafb20d3e93e","Type":"ContainerStarted","Data":"b979ac4de3b7822bb4ed880c2afbddb3040ff4dab47ace13648645cdb58e76f0"} Sep 09 23:08:34.764233 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.764212 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:34.764338 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.764264 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:34.779764 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.779737 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:34.814879 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:34.814830 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-km998" podStartSLOduration=9.224499766 podStartE2EDuration="26.814813926s" totalImagesPullingTime="17.59031416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:34.813236607 +0000 UTC m=+26.760648330" watchObservedRunningTime="2026-09-09 23:08:34.814813926 +0000 UTC m=+26.762225649" Sep 09 23:08:35.768338 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:35.768300 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecd37d40-2a05-4131-b63b-7dd542794f8a" containerID="49c5a585f2a024d4513518dd5ab7f2f0f210be6995f6e380d31eb5cdc927958c" exitCode=0 Sep 09 23:08:35.769053 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:35.768390 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerDied","Data":"49c5a585f2a024d4513518dd5ab7f2f0f210be6995f6e380d31eb5cdc927958c"} Sep 09 23:08:35.769129 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:35.769082 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:35.787041 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:35.787008 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:08:36.423038 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.422991 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s2tnq"] Sep 09 23:08:36.423241 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.423149 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:36.423420 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:36.423258 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:36.426408 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.426244 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qp8cl"] Sep 09 23:08:36.426546 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.426528 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:36.426666 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:36.426639 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:36.427176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.426990 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbcm2"] Sep 09 23:08:36.427176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.427081 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:36.427318 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:36.427193 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:36.773210 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.773101 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecd37d40-2a05-4131-b63b-7dd542794f8a" containerID="707647bdcaea4a0e5eba93ad04bd77c5fae8713aa59359aa44f406f180190d65" exitCode=0 Sep 09 23:08:36.773210 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:36.773155 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerDied","Data":"707647bdcaea4a0e5eba93ad04bd77c5fae8713aa59359aa44f406f180190d65"} Sep 09 23:08:37.544682 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:37.544644 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:37.544848 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:37.544763 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:37.780659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:37.780620 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecd37d40-2a05-4131-b63b-7dd542794f8a" containerID="f4aa3baa22979a4517c7e96beabf0b6881794eba9f4e56687d028500b0cf9bac" exitCode=0 Sep 09 23:08:37.781010 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:37.780662 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerDied","Data":"f4aa3baa22979a4517c7e96beabf0b6881794eba9f4e56687d028500b0cf9bac"} Sep 09 23:08:38.545059 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:38.544800 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:38.545244 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:38.544886 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:38.545244 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:38.545095 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:38.545244 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:38.545193 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:39.544207 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:39.544172 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:39.544629 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:39.544292 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:40.544084 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:40.544049 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:40.544250 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:40.544050 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:40.544250 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:40.544177 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qp8cl" podUID="b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea" Sep 09 23:08:40.544639 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:40.544615 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2tnq" podUID="e8364707-07d3-4a61-bcfc-f9af03c7ed0c" Sep 09 23:08:41.544469 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.544431 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:41.545017 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:41.544559 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:08:41.796251 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.796160 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-142.ec2.internal" event="NodeReady" Sep 09 23:08:41.796404 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.796318 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 23:08:41.830093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.830053 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cjdml"] Sep 09 23:08:41.849785 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.849751 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh"] Sep 09 23:08:41.849968 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.849947 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:41.854391 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.853830 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rn54l\"" Sep 09 23:08:41.854391 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.854209 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 23:08:41.854982 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.854957 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 23:08:41.859221 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.859198 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg"] Sep 09 23:08:41.859369 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.859356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.862425 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.862401 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 23:08:41.864769 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.864750 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 09 23:08:41.868005 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.867806 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2"] Sep 09 23:08:41.868005 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.867815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:41.876316 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.876282 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7464dc4644-c7hnn"] Sep 09 23:08:41.876468 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.876423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:41.879228 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.879202 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 23:08:41.882259 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.882225 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 09 23:08:41.882393 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.882333 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 09 23:08:41.882966 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.882690 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 09 23:08:41.884162 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.883937 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zbd44\"" Sep 09 23:08:41.884162 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.883945 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 23:08:41.884162 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.883977 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 23:08:41.884818 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.884649 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 09 23:08:41.886647 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.886625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cjdml"] Sep 09 23:08:41.886775 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.886664 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg"] Sep 09 23:08:41.886775 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.886683 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh"] Sep 09 23:08:41.886775 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.886702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7464dc4644-c7hnn"] Sep 09 23:08:41.886775 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.886726 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7vdvv"] Sep 09 23:08:41.887152 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.887132 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.894835 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.894798 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 23:08:41.894835 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.894802 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 23:08:41.895073 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.894968 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 23:08:41.895073 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.894996 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-48t55\"" Sep 09 23:08:41.904620 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.904596 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hgdps"] Sep 09 23:08:41.904779 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.904719 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:41.909166 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.909137 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 23:08:41.909374 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.909349 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4h26n\"" Sep 09 23:08:41.912814 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.912785 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 23:08:41.916427 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.916403 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 23:08:41.925540 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.925511 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2"] Sep 09 23:08:41.925692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.925564 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7vdvv"] Sep 09 23:08:41.925692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.925588 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hgdps"] Sep 09 23:08:41.925692 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.925650 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:41.927580 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.927554 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 23:08:41.928805 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.928778 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-249ln\"" Sep 09 23:08:41.929556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.929540 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 23:08:41.929800 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.929773 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 23:08:41.971017 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.970984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-certificates\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpq6s\" (UniqueName: \"kubernetes.io/projected/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-kube-api-access-fpq6s\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971079 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgkjc\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-kube-api-access-sgkjc\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971145 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971177 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-image-registry-private-configuration\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971205 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-trusted-ca\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-hub\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.971263 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971260 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vc6\" (UniqueName: \"kubernetes.io/projected/fee92304-2731-4bc2-804d-16b2319e131e-kube-api-access-h2vc6\") pod \"managed-serviceaccount-addon-agent-d7fbd5d84-d82hg\" (UID: \"fee92304-2731-4bc2-804d-16b2319e131e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:41.971627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pcf2\" (UniqueName: \"kubernetes.io/projected/08b1f98e-1a5b-4d04-ab10-570c9584e1bd-kube-api-access-7pcf2\") pod \"ingress-canary-7vdvv\" (UID: \"08b1f98e-1a5b-4d04-ab10-570c9584e1bd\") " pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:41.971627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-bound-sa-token\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-ca-trust-extracted\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971452 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-tmp\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:41.971627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-installation-pull-secrets\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971630 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-klusterlet-config\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971662 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cxv\" (UniqueName: \"kubernetes.io/projected/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-kube-api-access-l8cxv\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971683 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-additional-service-ca\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971715 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-ca\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fee92304-2731-4bc2-804d-16b2319e131e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7fbd5d84-d82hg\" (UID: \"fee92304-2731-4bc2-804d-16b2319e131e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971813 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08b1f98e-1a5b-4d04-ab10-570c9584e1bd-cert\") pod \"ingress-canary-7vdvv\" (UID: \"08b1f98e-1a5b-4d04-ab10-570c9584e1bd\") " pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:41.971839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:41.971834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-nginx-conf\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:42.072813 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.072717 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-certificates\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.072813 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.072764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpq6s\" (UniqueName: \"kubernetes.io/projected/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-kube-api-access-fpq6s\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.072813 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.072805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fad466-3949-46f2-b9cf-384cc72fc275-tmp-dir\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.073093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.072838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkjc\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-kube-api-access-sgkjc\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.072975 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.073093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-image-registry-private-configuration\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073053 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-trusted-ca\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-hub\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vc6\" (UniqueName: \"kubernetes.io/projected/fee92304-2731-4bc2-804d-16b2319e131e-kube-api-access-h2vc6\") pod \"managed-serviceaccount-addon-agent-d7fbd5d84-d82hg\" (UID: \"fee92304-2731-4bc2-804d-16b2319e131e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pcf2\" (UniqueName: \"kubernetes.io/projected/08b1f98e-1a5b-4d04-ab10-570c9584e1bd-kube-api-access-7pcf2\") pod \"ingress-canary-7vdvv\" (UID: \"08b1f98e-1a5b-4d04-ab10-570c9584e1bd\") " pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-bound-sa-token\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073222 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073270 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073294 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-ca-trust-extracted\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073314 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-tmp\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073336 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-installation-pull-secrets\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-klusterlet-config\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cxv\" (UniqueName: \"kubernetes.io/projected/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-kube-api-access-l8cxv\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073409 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-certificates\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073470 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-additional-service-ca\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.073493 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.073508 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-ca\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fee92304-2731-4bc2-804d-16b2319e131e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7fbd5d84-d82hg\" (UID: \"fee92304-2731-4bc2-804d-16b2319e131e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073575 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08b1f98e-1a5b-4d04-ab10-570c9584e1bd-cert\") pod \"ingress-canary-7vdvv\" (UID: \"08b1f98e-1a5b-4d04-ab10-570c9584e1bd\") " pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073602 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-nginx-conf\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073630 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99fad466-3949-46f2-b9cf-384cc72fc275-config-volume\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.073689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.073650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vxx\" (UniqueName: \"kubernetes.io/projected/99fad466-3949-46f2-b9cf-384cc72fc275-kube-api-access-w7vxx\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.074403 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.073935 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:08:42.573908054 +0000 UTC m=+34.521319755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:08:42.074403 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.074008 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:08:42.074403 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.074021 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-ca-trust-extracted\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.074403 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.074061 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:42.574049253 +0000 UTC m=+34.521460959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:08:42.074403 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.074321 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-additional-service-ca\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.074403 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.074368 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-tmp\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.076264 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.076032 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-nginx-conf\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:42.076264 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.076199 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-trusted-ca\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.079307 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.078793 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-installation-pull-secrets\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.079307 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.078793 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.079307 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.078858 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-klusterlet-config\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.079307 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.079242 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.079307 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.079265 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08b1f98e-1a5b-4d04-ab10-570c9584e1bd-cert\") pod \"ingress-canary-7vdvv\" (UID: \"08b1f98e-1a5b-4d04-ab10-570c9584e1bd\") " pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:42.079643 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.079354 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-ca\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.079800 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.079772 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-image-registry-private-configuration\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.080228 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.080170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-hub\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.080507 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.080489 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fee92304-2731-4bc2-804d-16b2319e131e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d7fbd5d84-d82hg\" (UID: \"fee92304-2731-4bc2-804d-16b2319e131e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:42.084868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.084825 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkjc\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-kube-api-access-sgkjc\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.085733 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.085675 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpq6s\" (UniqueName: \"kubernetes.io/projected/3bbaf163-83fa-4bda-85f2-25b6d3b1dbea-kube-api-access-fpq6s\") pod \"cluster-proxy-proxy-agent-746c74d9d-2nvwh\" (UID: \"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.086606 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.086560 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-bound-sa-token\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.086868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.086846 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cxv\" (UniqueName: \"kubernetes.io/projected/c8d5ed99-1e6d-485e-94be-d1bb5d22137f-kube-api-access-l8cxv\") pod \"klusterlet-addon-workmgr-cd56cb79b-cclc2\" (UID: \"c8d5ed99-1e6d-485e-94be-d1bb5d22137f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.087446 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.087424 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pcf2\" (UniqueName: \"kubernetes.io/projected/08b1f98e-1a5b-4d04-ab10-570c9584e1bd-kube-api-access-7pcf2\") pod \"ingress-canary-7vdvv\" (UID: \"08b1f98e-1a5b-4d04-ab10-570c9584e1bd\") " pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:42.087660 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.087639 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vc6\" (UniqueName: \"kubernetes.io/projected/fee92304-2731-4bc2-804d-16b2319e131e-kube-api-access-h2vc6\") pod \"managed-serviceaccount-addon-agent-d7fbd5d84-d82hg\" (UID: \"fee92304-2731-4bc2-804d-16b2319e131e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:42.174839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.174791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99fad466-3949-46f2-b9cf-384cc72fc275-config-volume\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.174839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.174847 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vxx\" (UniqueName: \"kubernetes.io/projected/99fad466-3949-46f2-b9cf-384cc72fc275-kube-api-access-w7vxx\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.175084 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.174888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fad466-3949-46f2-b9cf-384cc72fc275-tmp-dir\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.175084 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.174926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:42.175084 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.174961 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.175084 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.175071 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 23:08:42.175315 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.175129 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 23:08:42.175315 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.175144 2571 projected.go:196] Error preparing data for projected volume kube-api-access-z5s8q for pod openshift-network-diagnostics/network-check-target-s2tnq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:42.175315 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.175080 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:42.175315 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.175215 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q podName:e8364707-07d3-4a61-bcfc-f9af03c7ed0c nodeName:}" failed. No retries permitted until 2026-09-09 23:09:14.175195849 +0000 UTC m=+66.122607570 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z5s8q" (UniqueName: "kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q") pod "network-check-target-s2tnq" (UID: "e8364707-07d3-4a61-bcfc-f9af03c7ed0c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 23:08:42.175315 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.175247 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:08:42.67523068 +0000 UTC m=+34.622642380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:08:42.175315 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.175291 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99fad466-3949-46f2-b9cf-384cc72fc275-tmp-dir\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.175560 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.175502 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99fad466-3949-46f2-b9cf-384cc72fc275-config-volume\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.178794 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.178767 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:08:42.183907 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.183877 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vxx\" (UniqueName: \"kubernetes.io/projected/99fad466-3949-46f2-b9cf-384cc72fc275-kube-api-access-w7vxx\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.188637 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.188611 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:08:42.206440 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.206408 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:42.223367 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.223341 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7vdvv" Sep 09 23:08:42.275396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.275361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:42.275602 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.275441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:42.275602 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.275549 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:42.275602 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.275593 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:42.275736 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.275645 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:09:14.275623592 +0000 UTC m=+66.223035317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 23:08:42.275736 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.275662 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret podName:b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea nodeName:}" failed. No retries permitted until 2026-09-09 23:09:14.275655324 +0000 UTC m=+66.223067025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret") pod "global-pull-secret-syncer-qp8cl" (UID: "b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea") : object "kube-system"/"original-pull-secret" not registered Sep 09 23:08:42.544728 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.544696 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:08:42.545308 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.544700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:08:42.548051 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.548025 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-26qq8\"" Sep 09 23:08:42.548198 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.548063 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:08:42.548542 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.548348 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:08:42.548542 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.548382 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:08:42.578032 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.577996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:42.578236 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.578125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:42.578236 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.578170 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:08:42.578346 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.578236 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:08:42.578346 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.578248 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:08:42.578346 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.578252 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:43.578231293 +0000 UTC m=+35.525643009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:08:42.578346 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.578293 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:08:43.578277997 +0000 UTC m=+35.525689713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:08:42.679578 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:42.679537 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:42.679750 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.679694 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:42.679790 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:42.679760 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:08:43.679743257 +0000 UTC m=+35.627154977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:08:43.014454 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.014288 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg"] Sep 09 23:08:43.015212 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.015186 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2"] Sep 09 23:08:43.016398 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.016378 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7vdvv"] Sep 09 23:08:43.018606 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:43.018574 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d5ed99_1e6d_485e_94be_d1bb5d22137f.slice/crio-329cabf30fd8fccf6ec9c70d7481243328f4e1e674491323ce9e755d17c7b2f9 WatchSource:0}: Error finding container 329cabf30fd8fccf6ec9c70d7481243328f4e1e674491323ce9e755d17c7b2f9: Status 404 returned error can't find the container with id 329cabf30fd8fccf6ec9c70d7481243328f4e1e674491323ce9e755d17c7b2f9 Sep 09 23:08:43.018850 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:43.018828 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfee92304_2731_4bc2_804d_16b2319e131e.slice/crio-97a64f3e99e1cc773cc96c996c329c85e8156918c6cde4297ecb7863ac4f35b5 WatchSource:0}: Error finding container 97a64f3e99e1cc773cc96c996c329c85e8156918c6cde4297ecb7863ac4f35b5: Status 404 returned error can't find the container with id 97a64f3e99e1cc773cc96c996c329c85e8156918c6cde4297ecb7863ac4f35b5 Sep 09 23:08:43.019530 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:43.019506 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b1f98e_1a5b_4d04_ab10_570c9584e1bd.slice/crio-c15941311e278cec12b700856095870da82c773dd9ac09e62ce493ca4a426e5e WatchSource:0}: Error finding container c15941311e278cec12b700856095870da82c773dd9ac09e62ce493ca4a426e5e: Status 404 returned error can't find the container with id c15941311e278cec12b700856095870da82c773dd9ac09e62ce493ca4a426e5e Sep 09 23:08:43.019954 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.019920 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh"] Sep 09 23:08:43.022289 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:08:43.022267 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bbaf163_83fa_4bda_85f2_25b6d3b1dbea.slice/crio-517c5891e197507420ae8a67dc300bac5361cca378ae363538e695261687d8d4 WatchSource:0}: Error finding container 517c5891e197507420ae8a67dc300bac5361cca378ae363538e695261687d8d4: Status 404 returned error can't find the container with id 517c5891e197507420ae8a67dc300bac5361cca378ae363538e695261687d8d4 Sep 09 23:08:43.544590 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.544556 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:08:43.547706 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.547677 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:08:43.548260 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.547691 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hgrbx\"" Sep 09 23:08:43.586846 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.586803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:43.586846 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.586845 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:43.587056 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.586942 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:08:43.587056 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.586946 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:08:43.587056 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.586966 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:08:43.587056 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.586996 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:45.586983322 +0000 UTC m=+37.534395023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:08:43.587056 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.587011 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:08:45.587003611 +0000 UTC m=+37.534415313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:08:43.688149 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.688102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:43.688349 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.688295 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:43.688409 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:43.688375 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:08:45.68835387 +0000 UTC m=+37.635765576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:08:43.800927 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.800845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" event={"ID":"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea","Type":"ContainerStarted","Data":"517c5891e197507420ae8a67dc300bac5361cca378ae363538e695261687d8d4"} Sep 09 23:08:43.804400 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.804370 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecd37d40-2a05-4131-b63b-7dd542794f8a" containerID="cffc55b9e4072da374fd3ffc3973ca0fb40674207de95df86bda1d4ae2f0f918" exitCode=0 Sep 09 23:08:43.804556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.804456 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerDied","Data":"cffc55b9e4072da374fd3ffc3973ca0fb40674207de95df86bda1d4ae2f0f918"} Sep 09 23:08:43.805826 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.805800 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7vdvv" event={"ID":"08b1f98e-1a5b-4d04-ab10-570c9584e1bd","Type":"ContainerStarted","Data":"c15941311e278cec12b700856095870da82c773dd9ac09e62ce493ca4a426e5e"} Sep 09 23:08:43.807289 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.807267 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" event={"ID":"c8d5ed99-1e6d-485e-94be-d1bb5d22137f","Type":"ContainerStarted","Data":"329cabf30fd8fccf6ec9c70d7481243328f4e1e674491323ce9e755d17c7b2f9"} Sep 09 23:08:43.808733 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:43.808707 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" event={"ID":"fee92304-2731-4bc2-804d-16b2319e131e","Type":"ContainerStarted","Data":"97a64f3e99e1cc773cc96c996c329c85e8156918c6cde4297ecb7863ac4f35b5"} Sep 09 23:08:44.826253 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:44.826176 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecd37d40-2a05-4131-b63b-7dd542794f8a" containerID="79a90dc22feacca50b3e10bb0829774e285466f8aab2aefa4f6b549e11c95784" exitCode=0 Sep 09 23:08:44.826752 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:44.826300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerDied","Data":"79a90dc22feacca50b3e10bb0829774e285466f8aab2aefa4f6b549e11c95784"} Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:45.609421 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:45.609474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.609682 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.609751 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:49.609730581 +0000 UTC m=+41.557142307 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.610183 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.610199 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:08:45.610275 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.610243 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:08:49.610229608 +0000 UTC m=+41.557641312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:08:45.711132 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:45.710331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:45.711132 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.710623 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:45.711132 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:45.710759 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:08:49.710740072 +0000 UTC m=+41.658151778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:08:45.835216 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:45.835178 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sg94k" event={"ID":"ecd37d40-2a05-4131-b63b-7dd542794f8a","Type":"ContainerStarted","Data":"f2e67e59430d336571e072b28223001885e146afcc26a427723e65df3070db4d"} Sep 09 23:08:45.861760 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:45.861705 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sg94k" podStartSLOduration=8.781048647 podStartE2EDuration="37.861681697s" totalImagesPullingTime="29.08063305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:45.859701805 +0000 UTC m=+37.807113545" watchObservedRunningTime="2026-09-09 23:08:45.861681697 +0000 UTC m=+37.809093420" Sep 09 23:08:49.646287 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:49.646251 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:49.646287 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:49.646296 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:49.646707 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.646414 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:08:49.646707 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.646437 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:08:49.646707 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.646441 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:08:49.646707 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.646500 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:08:57.646478537 +0000 UTC m=+49.593890237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:08:49.646707 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.646519 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:08:57.646509643 +0000 UTC m=+49.593921344 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:08:49.747714 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:49.747671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:49.747889 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.747868 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:49.747991 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:49.747971 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:08:57.747931382 +0000 UTC m=+49.695343087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:08:52.855068 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.855025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" event={"ID":"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea","Type":"ContainerStarted","Data":"7459e58f3fed0cce268c96617db3b3a3847d545d4845da2145ff1545665975b2"} Sep 09 23:08:52.855068 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.855075 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" event={"ID":"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea","Type":"ContainerStarted","Data":"b7452cc49201bd55f1b86e07f0b26a77f3770af6280a0e0ed5a3cfe8e5ed4e66"} Sep 09 23:08:52.855570 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.855089 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" event={"ID":"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea","Type":"ContainerStarted","Data":"13d8fbca8e467a9374d82ed6c3d10a96389911b5862bccfa6429fc003acf9312"} Sep 09 23:08:52.856547 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.856526 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7vdvv" event={"ID":"08b1f98e-1a5b-4d04-ab10-570c9584e1bd","Type":"ContainerStarted","Data":"f4fab7e6721d2237379275cb249cc0defe381eb16e1958fcbf90530831d04af9"} Sep 09 23:08:52.857930 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.857907 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" event={"ID":"c8d5ed99-1e6d-485e-94be-d1bb5d22137f","Type":"ContainerStarted","Data":"9bedf78a4d0b0ade47e08b9f8f75476782f35a552b45d3678b168a70b8bc7091"} Sep 09 23:08:52.858030 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.857949 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:52.859433 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.859403 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" event={"ID":"fee92304-2731-4bc2-804d-16b2319e131e","Type":"ContainerStarted","Data":"e0456eddf7e5f2b207676f4a534c0d0996e03a83b4824bf549ece905679603ca"} Sep 09 23:08:52.860071 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.860052 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:08:52.878356 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.878310 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" podStartSLOduration=26.869163572 podStartE2EDuration="35.878294605s" totalImagesPullingTime="9.009131033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:52.877640621 +0000 UTC m=+44.825052344" watchObservedRunningTime="2026-09-09 23:08:52.878294605 +0000 UTC m=+44.825706327" Sep 09 23:08:52.895379 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.895334 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" podStartSLOduration=26.883213041 podStartE2EDuration="35.895319543s" totalImagesPullingTime="9.012106502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:52.895156588 +0000 UTC m=+44.842568311" watchObservedRunningTime="2026-09-09 23:08:52.895319543 +0000 UTC m=+44.842731266" Sep 09 23:08:52.931519 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.931467 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" podStartSLOduration=26.911862594 podStartE2EDuration="35.931449516s" totalImagesPullingTime="9.019586922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:52.930709731 +0000 UTC m=+44.878121454" watchObservedRunningTime="2026-09-09 23:08:52.931449516 +0000 UTC m=+44.878861238" Sep 09 23:08:52.931684 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:52.931547 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7vdvv" podStartSLOduration=2.92099924 podStartE2EDuration="11.931541591s" totalImagesPullingTime="9.010542351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:08:52.913653234 +0000 UTC m=+44.861064957" watchObservedRunningTime="2026-09-09 23:08:52.931541591 +0000 UTC m=+44.878953313" Sep 09 23:08:57.715331 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:57.715286 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:08:57.715331 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:57.715331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:08:57.715875 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.715478 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:08:57.715875 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.715509 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:08:57.715875 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.715563 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:08:57.715875 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.715596 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:09:13.715573947 +0000 UTC m=+65.662985665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:08:57.715875 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.715614 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:09:13.715605136 +0000 UTC m=+65.663016861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:08:57.816588 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:08:57.816551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:08:57.816773 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.816693 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:08:57.816773 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:08:57.816763 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:09:13.816748407 +0000 UTC m=+65.764160108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:09:07.791804 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:07.791768 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-km998" Sep 09 23:09:13.738708 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:13.738657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:09:13.738708 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:13.738710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:09:13.739291 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.738812 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:09:13.739291 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.738821 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:09:13.739291 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.738841 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:09:13.739291 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.738891 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:09:45.738871809 +0000 UTC m=+97.686283509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:09:13.739291 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.738905 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:09:45.738898997 +0000 UTC m=+97.686310699 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:09:13.839455 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:13.839411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:09:13.839654 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.839555 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:09:13.839654 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:13.839631 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:09:45.839605967 +0000 UTC m=+97.787017668 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:09:14.241426 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.241379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:09:14.244720 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.244694 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 23:09:14.254993 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.254962 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 23:09:14.265578 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.265543 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5s8q\" (UniqueName: \"kubernetes.io/projected/e8364707-07d3-4a61-bcfc-f9af03c7ed0c-kube-api-access-z5s8q\") pod \"network-check-target-s2tnq\" (UID: \"e8364707-07d3-4a61-bcfc-f9af03c7ed0c\") " pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:09:14.342089 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.342049 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:09:14.342271 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.342127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:09:14.345585 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.345560 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 23:09:14.345681 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.345563 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 23:09:14.353144 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:14.353098 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:09:14.353224 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:14.353212 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:10:18.353195819 +0000 UTC m=+130.300607521 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : secret "metrics-daemon-secret" not found Sep 09 23:09:14.355414 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.355390 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea-original-pull-secret\") pod \"global-pull-secret-syncer-qp8cl\" (UID: \"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea\") " pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:09:14.359145 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.359102 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-26qq8\"" Sep 09 23:09:14.363771 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.363745 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qp8cl" Sep 09 23:09:14.366884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.366860 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:09:14.663336 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.663300 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qp8cl"] Sep 09 23:09:14.666776 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:09:14.666747 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bffd7d_c4a7_4d1e_a5f4_a808ab84e0ea.slice/crio-17a17f5be542f418aba325c2d6b18e019b63c7ba64ef3ecebfbb94f0d9533ff6 WatchSource:0}: Error finding container 17a17f5be542f418aba325c2d6b18e019b63c7ba64ef3ecebfbb94f0d9533ff6: Status 404 returned error can't find the container with id 17a17f5be542f418aba325c2d6b18e019b63c7ba64ef3ecebfbb94f0d9533ff6 Sep 09 23:09:14.690765 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.690742 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s2tnq"] Sep 09 23:09:14.693081 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:09:14.693039 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8364707_07d3_4a61_bcfc_f9af03c7ed0c.slice/crio-38f93cd412652c71402828139635ffd977d3c43dd32480ef004b157be9f89005 WatchSource:0}: Error finding container 38f93cd412652c71402828139635ffd977d3c43dd32480ef004b157be9f89005: Status 404 returned error can't find the container with id 38f93cd412652c71402828139635ffd977d3c43dd32480ef004b157be9f89005 Sep 09 23:09:14.922663 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.922561 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qp8cl" event={"ID":"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea","Type":"ContainerStarted","Data":"17a17f5be542f418aba325c2d6b18e019b63c7ba64ef3ecebfbb94f0d9533ff6"} Sep 09 23:09:14.923710 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:14.923681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s2tnq" event={"ID":"e8364707-07d3-4a61-bcfc-f9af03c7ed0c","Type":"ContainerStarted","Data":"38f93cd412652c71402828139635ffd977d3c43dd32480ef004b157be9f89005"} Sep 09 23:09:19.941646 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:19.941606 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s2tnq" event={"ID":"e8364707-07d3-4a61-bcfc-f9af03c7ed0c","Type":"ContainerStarted","Data":"d0c08f35cc5f1b65ef27769db30816eddfdfc35b72fe4b5278c317edd4b6892a"} Sep 09 23:09:19.942087 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:19.941724 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:09:19.943189 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:19.943165 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qp8cl" event={"ID":"b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea","Type":"ContainerStarted","Data":"2434e811768db1d1eb1be16464930d0aee20f3964330e251a9f2c640a284c3f1"} Sep 09 23:09:19.958520 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:19.958467 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s2tnq" podStartSLOduration=67.458247123 podStartE2EDuration="1m11.958450497s" totalImagesPullingTime="4.500203374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:19.958033079 +0000 UTC m=+71.905444802" watchObservedRunningTime="2026-09-09 23:09:19.958450497 +0000 UTC m=+71.905862219" Sep 09 23:09:19.973783 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:19.973726 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qp8cl" podStartSLOduration=67.442608061 podStartE2EDuration="1m11.973703144s" totalImagesPullingTime="4.531095083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:09:19.973358845 +0000 UTC m=+71.920770568" watchObservedRunningTime="2026-09-09 23:09:19.973703144 +0000 UTC m=+71.921114871" Sep 09 23:09:45.794975 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:45.794935 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:09:45.794975 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:45.794986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:09:45.795572 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.795090 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:09:45.795572 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.795127 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:09:45.795572 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.795158 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:09:45.795572 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.795196 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:10:49.795178322 +0000 UTC m=+161.742590027 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:09:45.795572 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.795221 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:10:49.795205517 +0000 UTC m=+161.742617238 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:09:45.895559 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:45.895521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:09:45.895744 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.895678 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:09:45.895786 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:09:45.895752 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:10:49.89573701 +0000 UTC m=+161.843148711 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:09:50.950518 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:09:50.950489 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s2tnq" Sep 09 23:10:18.444494 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:18.444430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:10:18.445005 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:18.444592 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:10:18.445005 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:18.444681 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:12:20.444663477 +0000 UTC m=+252.392075183 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : secret "metrics-daemon-secret" not found Sep 09 23:10:44.864311 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:44.864266 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" podUID="9f2a5c74-93c8-486e-a372-c4223fe2ac2f" Sep 09 23:10:44.917179 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:44.917095 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" Sep 09 23:10:44.935666 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:44.935613 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hgdps" podUID="99fad466-3949-46f2-b9cf-384cc72fc275" Sep 09 23:10:45.195319 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:45.195231 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hgdps" Sep 09 23:10:45.195488 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:45.195241 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:10:46.553862 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:46.553820 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:10:49.879456 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:49.879406 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:10:49.879456 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:49.879455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:10:49.880172 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.879598 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 23:10:49.880172 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.879628 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7464dc4644-c7hnn: secret "image-registry-tls" not found Sep 09 23:10:49.880172 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.879660 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 23:10:49.880172 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.879702 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls podName:f20e2dbe-2a4e-4252-ab29-2dc666c8155d nodeName:}" failed. No retries permitted until 2026-09-09 23:12:51.879679917 +0000 UTC m=+283.827091631 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls") pod "image-registry-7464dc4644-c7hnn" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d") : secret "image-registry-tls" not found Sep 09 23:10:49.880172 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.879722 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert podName:9f2a5c74-93c8-486e-a372-c4223fe2ac2f nodeName:}" failed. No retries permitted until 2026-09-09 23:12:51.879711952 +0000 UTC m=+283.827123656 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-cjdml" (UID: "9f2a5c74-93c8-486e-a372-c4223fe2ac2f") : secret "networking-console-plugin-cert" not found Sep 09 23:10:49.980477 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:49.980431 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:10:49.980668 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.980580 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 23:10:49.980668 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:10:49.980648 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls podName:99fad466-3949-46f2-b9cf-384cc72fc275 nodeName:}" failed. No retries permitted until 2026-09-09 23:12:51.980631288 +0000 UTC m=+283.928043015 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls") pod "dns-default-hgdps" (UID: "99fad466-3949-46f2-b9cf-384cc72fc275") : secret "dns-default-metrics-tls" not found Sep 09 23:10:52.859140 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:52.859077 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" podUID="c8d5ed99-1e6d-485e-94be-d1bb5d22137f" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused" Sep 09 23:10:53.219317 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:53.219231 2571 generic.go:356] "Generic (PLEG): container finished" podID="c8d5ed99-1e6d-485e-94be-d1bb5d22137f" containerID="9bedf78a4d0b0ade47e08b9f8f75476782f35a552b45d3678b168a70b8bc7091" exitCode=1 Sep 09 23:10:53.219486 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:53.219307 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" event={"ID":"c8d5ed99-1e6d-485e-94be-d1bb5d22137f","Type":"ContainerDied","Data":"9bedf78a4d0b0ade47e08b9f8f75476782f35a552b45d3678b168a70b8bc7091"} Sep 09 23:10:53.219696 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:53.219678 2571 scope.go:117] "RemoveContainer" containerID="9bedf78a4d0b0ade47e08b9f8f75476782f35a552b45d3678b168a70b8bc7091" Sep 09 23:10:53.220798 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:53.220770 2571 generic.go:356] "Generic (PLEG): container finished" podID="fee92304-2731-4bc2-804d-16b2319e131e" containerID="e0456eddf7e5f2b207676f4a534c0d0996e03a83b4824bf549ece905679603ca" exitCode=255 Sep 09 23:10:53.220882 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:53.220843 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" event={"ID":"fee92304-2731-4bc2-804d-16b2319e131e","Type":"ContainerDied","Data":"e0456eddf7e5f2b207676f4a534c0d0996e03a83b4824bf549ece905679603ca"} Sep 09 23:10:53.221138 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:53.221106 2571 scope.go:117] "RemoveContainer" containerID="e0456eddf7e5f2b207676f4a534c0d0996e03a83b4824bf549ece905679603ca" Sep 09 23:10:54.225940 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:54.225906 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" event={"ID":"c8d5ed99-1e6d-485e-94be-d1bb5d22137f","Type":"ContainerStarted","Data":"3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173"} Sep 09 23:10:54.226377 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:54.226222 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:10:54.226884 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:54.226863 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:10:54.227677 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:54.227655 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" event={"ID":"fee92304-2731-4bc2-804d-16b2319e131e","Type":"ContainerStarted","Data":"4e6e7c5633eb594960b3bda1be91b05da1f6c2675ecf3464954202a99666a2db"} Sep 09 23:10:58.545529 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:58.545477 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:10:59.544489 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:10:59.544441 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:12:18.489983 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.489949 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9lrfw"] Sep 09 23:12:18.492971 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.492955 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.495718 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.495681 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 23:12:18.495820 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.495800 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 23:12:18.495888 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.495872 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 23:12:18.497082 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.497065 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xkjxb\"" Sep 09 23:12:18.497190 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.497103 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 23:12:18.504387 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.504363 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lrfw"] Sep 09 23:12:18.630807 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.630756 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.630999 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.630832 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xss\" (UniqueName: \"kubernetes.io/projected/ce94d45c-9b72-48ec-8019-2f744435478e-kube-api-access-m7xss\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.630999 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.630879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce94d45c-9b72-48ec-8019-2f744435478e-data-volume\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.631138 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.631042 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.631203 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.631131 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce94d45c-9b72-48ec-8019-2f744435478e-crio-socket\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732244 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732192 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732244 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732254 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce94d45c-9b72-48ec-8019-2f744435478e-crio-socket\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732277 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732310 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xss\" (UniqueName: \"kubernetes.io/projected/ce94d45c-9b72-48ec-8019-2f744435478e-kube-api-access-m7xss\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce94d45c-9b72-48ec-8019-2f744435478e-data-volume\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732463 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:18.732338 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:12:18.732463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732418 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ce94d45c-9b72-48ec-8019-2f744435478e-crio-socket\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.732463 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:18.732424 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:12:18.732644 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:18.732429 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:19.232408829 +0000 UTC m=+251.179820531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : configmap "kube-rbac-proxy" not found Sep 09 23:12:18.732644 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:18.732509 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:19.232490821 +0000 UTC m=+251.179902530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : secret "insights-runtime-extractor-tls" not found Sep 09 23:12:18.732716 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.732650 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce94d45c-9b72-48ec-8019-2f744435478e-data-volume\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:18.741498 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:18.741440 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xss\" (UniqueName: \"kubernetes.io/projected/ce94d45c-9b72-48ec-8019-2f744435478e-kube-api-access-m7xss\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:19.236917 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:19.236875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:19.237110 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:19.236931 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:19.237110 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:19.237003 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:12:19.237110 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:19.237019 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:12:19.237110 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:19.237069 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:20.237055222 +0000 UTC m=+252.184466924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : configmap "kube-rbac-proxy" not found Sep 09 23:12:19.237301 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:19.237108 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:20.237092214 +0000 UTC m=+252.184503915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : secret "insights-runtime-extractor-tls" not found Sep 09 23:12:20.247647 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:20.247601 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:20.248054 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:20.247665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:20.248054 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:20.247721 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:12:20.248054 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:20.247760 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:12:20.248054 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:20.247789 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:22.247774095 +0000 UTC m=+254.195185800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : configmap "kube-rbac-proxy" not found Sep 09 23:12:20.248054 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:20.247810 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:22.24779847 +0000 UTC m=+254.195210176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : secret "insights-runtime-extractor-tls" not found Sep 09 23:12:20.448722 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:20.448685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:12:20.448906 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:20.448827 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 23:12:20.448906 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:20.448890 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs podName:8647901c-c88a-4922-a37b-ecff32c52f6f nodeName:}" failed. No retries permitted until 2026-09-09 23:14:22.448874827 +0000 UTC m=+374.396286532 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs") pod "network-metrics-daemon-xbcm2" (UID: "8647901c-c88a-4922-a37b-ecff32c52f6f") : secret "metrics-daemon-secret" not found Sep 09 23:12:20.614969 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:20.614939 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tg9zt_2ae6772e-e29f-49ed-b872-0c2f6b30ac90/dns-node-resolver/0.log" Sep 09 23:12:21.616033 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:21.616003 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v7v25_86cd631b-4036-4398-b0a6-63d6c08bb6b8/node-ca/0.log" Sep 09 23:12:22.024462 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:22.024392 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7vdvv_08b1f98e-1a5b-4d04-ab10-570c9584e1bd/serve-healthcheck-canary/0.log" Sep 09 23:12:22.264034 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:22.263987 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:22.264034 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:22.264050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:22.264288 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:22.264151 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:12:22.264288 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:22.264167 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:12:22.264288 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:22.264213 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:26.264198472 +0000 UTC m=+258.211610173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : configmap "kube-rbac-proxy" not found Sep 09 23:12:22.264288 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:22.264229 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:26.264221065 +0000 UTC m=+258.211632766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : secret "insights-runtime-extractor-tls" not found Sep 09 23:12:26.296435 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:26.296374 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:26.296910 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:26.296474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:26.296910 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:26.296529 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:12:26.296910 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:26.296600 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:34.296583804 +0000 UTC m=+266.243995510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : secret "insights-runtime-extractor-tls" not found Sep 09 23:12:26.296910 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:26.296533 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 23:12:26.296910 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:26.296675 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:34.29666245 +0000 UTC m=+266.244074151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : configmap "kube-rbac-proxy" not found Sep 09 23:12:32.180291 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:32.180207 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" podUID="3bbaf163-83fa-4bda-85f2-25b6d3b1dbea" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 23:12:34.358208 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:34.358165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:34.358623 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:34.358230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:34.358623 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:34.358330 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 23:12:34.358623 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:34.358411 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls podName:ce94d45c-9b72-48ec-8019-2f744435478e nodeName:}" failed. No retries permitted until 2026-09-09 23:12:50.358394313 +0000 UTC m=+282.305806018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lrfw" (UID: "ce94d45c-9b72-48ec-8019-2f744435478e") : secret "insights-runtime-extractor-tls" not found Sep 09 23:12:34.358738 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:34.358707 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ce94d45c-9b72-48ec-8019-2f744435478e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:42.180413 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:42.180372 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" podUID="3bbaf163-83fa-4bda-85f2-25b6d3b1dbea" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 23:12:48.196343 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:48.196289 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" podUID="9f2a5c74-93c8-486e-a372-c4223fe2ac2f" Sep 09 23:12:48.196343 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:48.196290 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hgdps" podUID="99fad466-3949-46f2-b9cf-384cc72fc275" Sep 09 23:12:48.553754 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:48.553723 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:12:48.553980 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:48.553723 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hgdps" Sep 09 23:12:50.383889 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:50.383849 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:50.386176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:50.386155 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ce94d45c-9b72-48ec-8019-2f744435478e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lrfw\" (UID: \"ce94d45c-9b72-48ec-8019-2f744435478e\") " pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:50.601820 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:50.601777 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lrfw" Sep 09 23:12:50.870006 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:50.869975 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lrfw"] Sep 09 23:12:50.873071 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:12:50.873041 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce94d45c_9b72_48ec_8019_2f744435478e.slice/crio-e86fcec27af71d796d1b4b58aebb43f45c4a9246d3335b9afeadb39e6397c798 WatchSource:0}: Error finding container e86fcec27af71d796d1b4b58aebb43f45c4a9246d3335b9afeadb39e6397c798: Status 404 returned error can't find the container with id e86fcec27af71d796d1b4b58aebb43f45c4a9246d3335b9afeadb39e6397c798 Sep 09 23:12:51.564128 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.564084 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lrfw" event={"ID":"ce94d45c-9b72-48ec-8019-2f744435478e","Type":"ContainerStarted","Data":"3ff90a251302a2ee43302789755e658af0ca10fe840ec234cee8f47587720ba8"} Sep 09 23:12:51.564606 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.564172 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lrfw" event={"ID":"ce94d45c-9b72-48ec-8019-2f744435478e","Type":"ContainerStarted","Data":"e86fcec27af71d796d1b4b58aebb43f45c4a9246d3335b9afeadb39e6397c798"} Sep 09 23:12:51.896218 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.896090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:12:51.896218 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.896158 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:12:51.898619 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.898594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f2a5c74-93c8-486e-a372-c4223fe2ac2f-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-cjdml\" (UID: \"9f2a5c74-93c8-486e-a372-c4223fe2ac2f\") " pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:12:51.898714 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.898594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"image-registry-7464dc4644-c7hnn\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:12:51.996975 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.996922 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:12:51.999275 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:51.999245 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99fad466-3949-46f2-b9cf-384cc72fc275-metrics-tls\") pod \"dns-default-hgdps\" (UID: \"99fad466-3949-46f2-b9cf-384cc72fc275\") " pod="openshift-dns/dns-default-hgdps" Sep 09 23:12:52.048842 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.048806 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-48t55\"" Sep 09 23:12:52.056056 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.056031 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:12:52.157739 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.157657 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rn54l\"" Sep 09 23:12:52.157739 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.157657 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-249ln\"" Sep 09 23:12:52.164933 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.164907 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" Sep 09 23:12:52.165034 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.165013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hgdps" Sep 09 23:12:52.180182 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.180148 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" podUID="3bbaf163-83fa-4bda-85f2-25b6d3b1dbea" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 09 23:12:52.180309 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.180223 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" Sep 09 23:12:52.180731 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.180699 2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"7459e58f3fed0cce268c96617db3b3a3847d545d4845da2145ff1545665975b2"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 09 23:12:52.180824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.180750 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" podUID="3bbaf163-83fa-4bda-85f2-25b6d3b1dbea" containerName="service-proxy" containerID="cri-o://7459e58f3fed0cce268c96617db3b3a3847d545d4845da2145ff1545665975b2" gracePeriod=30 Sep 09 23:12:52.327847 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.327750 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7464dc4644-c7hnn"] Sep 09 23:12:52.331618 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:12:52.331587 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20e2dbe_2a4e_4252_ab29_2dc666c8155d.slice/crio-2522783863d9af7b4b1e2fccf066275e7cb8ddfb60f84b8b03f0a6d33f0ce826 WatchSource:0}: Error finding container 2522783863d9af7b4b1e2fccf066275e7cb8ddfb60f84b8b03f0a6d33f0ce826: Status 404 returned error can't find the container with id 2522783863d9af7b4b1e2fccf066275e7cb8ddfb60f84b8b03f0a6d33f0ce826 Sep 09 23:12:52.440053 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.439951 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-cjdml"] Sep 09 23:12:52.443165 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.443075 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hgdps"] Sep 09 23:12:52.443486 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:12:52.443451 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2a5c74_93c8_486e_a372_c4223fe2ac2f.slice/crio-033889334beb68975b6ed81afb59b961046301bac1b0eacbc97203271169c194 WatchSource:0}: Error finding container 033889334beb68975b6ed81afb59b961046301bac1b0eacbc97203271169c194: Status 404 returned error can't find the container with id 033889334beb68975b6ed81afb59b961046301bac1b0eacbc97203271169c194 Sep 09 23:12:52.446377 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:12:52.446307 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fad466_3949_46f2_b9cf_384cc72fc275.slice/crio-3942ed67bf9a8cc4cf3e79ff818ba28e7178344287bdec93640f90ebefde7f38 WatchSource:0}: Error finding container 3942ed67bf9a8cc4cf3e79ff818ba28e7178344287bdec93640f90ebefde7f38: Status 404 returned error can't find the container with id 3942ed67bf9a8cc4cf3e79ff818ba28e7178344287bdec93640f90ebefde7f38 Sep 09 23:12:52.570593 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.570552 2571 generic.go:356] "Generic (PLEG): container finished" podID="3bbaf163-83fa-4bda-85f2-25b6d3b1dbea" containerID="7459e58f3fed0cce268c96617db3b3a3847d545d4845da2145ff1545665975b2" exitCode=2 Sep 09 23:12:52.571040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.570613 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" event={"ID":"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea","Type":"ContainerDied","Data":"7459e58f3fed0cce268c96617db3b3a3847d545d4845da2145ff1545665975b2"} Sep 09 23:12:52.571040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.570660 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-746c74d9d-2nvwh" event={"ID":"3bbaf163-83fa-4bda-85f2-25b6d3b1dbea","Type":"ContainerStarted","Data":"78f7bd9006c4782dd19bc9a5f0f216250e49942f3ed38e5b25e1ab03e1b01639"} Sep 09 23:12:52.572099 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.572055 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hgdps" event={"ID":"99fad466-3949-46f2-b9cf-384cc72fc275","Type":"ContainerStarted","Data":"3942ed67bf9a8cc4cf3e79ff818ba28e7178344287bdec93640f90ebefde7f38"} Sep 09 23:12:52.574256 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.574221 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lrfw" event={"ID":"ce94d45c-9b72-48ec-8019-2f744435478e","Type":"ContainerStarted","Data":"dcd63bc42ac3e1db6e68d489e5fe4b01c7d43f202c8889fac4dddf107692af2c"} Sep 09 23:12:52.575678 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.575656 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" event={"ID":"9f2a5c74-93c8-486e-a372-c4223fe2ac2f","Type":"ContainerStarted","Data":"033889334beb68975b6ed81afb59b961046301bac1b0eacbc97203271169c194"} Sep 09 23:12:52.577528 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.577507 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" event={"ID":"f20e2dbe-2a4e-4252-ab29-2dc666c8155d","Type":"ContainerStarted","Data":"c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c"} Sep 09 23:12:52.577625 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.577547 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" event={"ID":"f20e2dbe-2a4e-4252-ab29-2dc666c8155d","Type":"ContainerStarted","Data":"2522783863d9af7b4b1e2fccf066275e7cb8ddfb60f84b8b03f0a6d33f0ce826"} Sep 09 23:12:52.577996 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.577976 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:12:52.610038 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:52.609966 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" podStartSLOduration=284.609943964 podStartE2EDuration="4m44.609943964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:52.607706315 +0000 UTC m=+284.555118039" watchObservedRunningTime="2026-09-09 23:12:52.609943964 +0000 UTC m=+284.557355688" Sep 09 23:12:53.585730 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.585692 2571 generic.go:356] "Generic (PLEG): container finished" podID="c8d5ed99-1e6d-485e-94be-d1bb5d22137f" containerID="3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173" exitCode=1 Sep 09 23:12:53.585730 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.585725 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" event={"ID":"c8d5ed99-1e6d-485e-94be-d1bb5d22137f","Type":"ContainerDied","Data":"3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173"} Sep 09 23:12:53.586301 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.585820 2571 scope.go:117] "RemoveContainer" containerID="9bedf78a4d0b0ade47e08b9f8f75476782f35a552b45d3678b168a70b8bc7091" Sep 09 23:12:53.586301 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.586220 2571 scope.go:117] "RemoveContainer" containerID="3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173" Sep 09 23:12:53.586422 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:53.586396 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-cd56cb79b-cclc2_open-cluster-management-agent-addon(c8d5ed99-1e6d-485e-94be-d1bb5d22137f)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" podUID="c8d5ed99-1e6d-485e-94be-d1bb5d22137f" Sep 09 23:12:53.588736 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.588618 2571 generic.go:356] "Generic (PLEG): container finished" podID="fee92304-2731-4bc2-804d-16b2319e131e" containerID="4e6e7c5633eb594960b3bda1be91b05da1f6c2675ecf3464954202a99666a2db" exitCode=255 Sep 09 23:12:53.588847 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.588727 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" event={"ID":"fee92304-2731-4bc2-804d-16b2319e131e","Type":"ContainerDied","Data":"4e6e7c5633eb594960b3bda1be91b05da1f6c2675ecf3464954202a99666a2db"} Sep 09 23:12:53.589095 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.589076 2571 scope.go:117] "RemoveContainer" containerID="4e6e7c5633eb594960b3bda1be91b05da1f6c2675ecf3464954202a99666a2db" Sep 09 23:12:53.589367 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:53.589329 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-d7fbd5d84-d82hg_open-cluster-management-agent-addon(fee92304-2731-4bc2-804d-16b2319e131e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" podUID="fee92304-2731-4bc2-804d-16b2319e131e" Sep 09 23:12:53.591800 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.591774 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lrfw" event={"ID":"ce94d45c-9b72-48ec-8019-2f744435478e","Type":"ContainerStarted","Data":"b3a9bb34427ecb653be8361effecb8fb958fd96d60b9f6d1e43aa5a6423fb98d"} Sep 09 23:12:53.636362 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.636303 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9lrfw" podStartSLOduration=33.505733776 podStartE2EDuration="35.636286839s" totalImagesPullingTime="2.130553063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:12:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:53.63558653 +0000 UTC m=+285.582998255" watchObservedRunningTime="2026-09-09 23:12:53.636286839 +0000 UTC m=+285.583698556" Sep 09 23:12:53.855764 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:53.855736 2571 scope.go:117] "RemoveContainer" containerID="e0456eddf7e5f2b207676f4a534c0d0996e03a83b4824bf549ece905679603ca" Sep 09 23:12:54.226507 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:54.226423 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:12:54.596755 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:54.596729 2571 scope.go:117] "RemoveContainer" containerID="3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173" Sep 09 23:12:54.597181 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:54.596910 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-cd56cb79b-cclc2_open-cluster-management-agent-addon(c8d5ed99-1e6d-485e-94be-d1bb5d22137f)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" podUID="c8d5ed99-1e6d-485e-94be-d1bb5d22137f" Sep 09 23:12:54.599129 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:54.599092 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" event={"ID":"9f2a5c74-93c8-486e-a372-c4223fe2ac2f","Type":"ContainerStarted","Data":"000251b516688c414ab257afe3f0f419deb9efb0be8ed57856299e77e96040d3"} Sep 09 23:12:54.629648 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:54.629600 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-cjdml" podStartSLOduration=275.181374711 podStartE2EDuration="4m36.629579843s" totalImagesPullingTime="1.448205132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:54.629277667 +0000 UTC m=+286.576689402" watchObservedRunningTime="2026-09-09 23:12:54.629579843 +0000 UTC m=+286.576991563" Sep 09 23:12:55.604240 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:55.604204 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hgdps" event={"ID":"99fad466-3949-46f2-b9cf-384cc72fc275","Type":"ContainerStarted","Data":"138de793118c12a6f208c8e6c3d9202af47e1ddd59fc1f97342b44e6b6a3f103"} Sep 09 23:12:55.604625 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:55.604256 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hgdps" event={"ID":"99fad466-3949-46f2-b9cf-384cc72fc275","Type":"ContainerStarted","Data":"110d4238c40bc8ad27e979f3e2cafad18d8c740baf3f0aab0adc736a1ab31e8f"} Sep 09 23:12:55.623679 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:55.623627 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hgdps" podStartSLOduration=252.555209879 podStartE2EDuration="4m14.623610402s" totalImagesPullingTime="2.068400523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:12:55.623216888 +0000 UTC m=+287.570628610" watchObservedRunningTime="2026-09-09 23:12:55.623610402 +0000 UTC m=+287.571022124" Sep 09 23:12:56.608060 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.608024 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hgdps" Sep 09 23:12:56.900619 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.900533 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7tn22"] Sep 09 23:12:56.905521 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.905489 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.908277 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.908245 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 23:12:56.908436 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.908289 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 23:12:56.908531 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.908514 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hzzbt\"" Sep 09 23:12:56.908596 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.908516 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 23:12:56.908596 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.908517 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 23:12:56.909619 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.909600 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 23:12:56.909697 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.909649 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 23:12:56.934449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlxs\" (UniqueName: \"kubernetes.io/projected/d5e895e5-ef52-4938-8fcf-e46232bc95a7-kube-api-access-7vlxs\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934449 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934457 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-accelerators-collector-config\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934482 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-tls\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934548 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e895e5-ef52-4938-8fcf-e46232bc95a7-metrics-client-ca\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934583 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-root\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934633 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-textfile\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934814 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934673 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-sys\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:56.934814 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:56.934690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-wtmp\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035348 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035306 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-tls\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035360 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e895e5-ef52-4938-8fcf-e46232bc95a7-metrics-client-ca\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035383 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035401 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-root\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035425 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-textfile\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-sys\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-wtmp\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlxs\" (UniqueName: \"kubernetes.io/projected/d5e895e5-ef52-4938-8fcf-e46232bc95a7-kube-api-access-7vlxs\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:57.035483 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035510 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-accelerators-collector-config\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035516 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-sys\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035483 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-root\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.035556 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:12:57.035560 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-tls podName:d5e895e5-ef52-4938-8fcf-e46232bc95a7 nodeName:}" failed. No retries permitted until 2026-09-09 23:12:57.535539054 +0000 UTC m=+289.482950770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-tls") pod "node-exporter-7tn22" (UID: "d5e895e5-ef52-4938-8fcf-e46232bc95a7") : secret "node-exporter-tls" not found Sep 09 23:12:57.036040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035647 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-wtmp\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.036040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035858 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-textfile\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.036040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.035969 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5e895e5-ef52-4938-8fcf-e46232bc95a7-metrics-client-ca\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.036040 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.036029 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-accelerators-collector-config\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.037719 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.037698 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.044725 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.044695 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlxs\" (UniqueName: \"kubernetes.io/projected/d5e895e5-ef52-4938-8fcf-e46232bc95a7-kube-api-access-7vlxs\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.538735 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.538685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-tls\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.541056 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.541036 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d5e895e5-ef52-4938-8fcf-e46232bc95a7-node-exporter-tls\") pod \"node-exporter-7tn22\" (UID: \"d5e895e5-ef52-4938-8fcf-e46232bc95a7\") " pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.814911 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:57.814869 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7tn22" Sep 09 23:12:57.825278 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:12:57.825247 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e895e5_ef52_4938_8fcf_e46232bc95a7.slice/crio-69a2a3308a45275d726bcc3619198090ed49a286b7e2b6b3be5f04abb4d3539c WatchSource:0}: Error finding container 69a2a3308a45275d726bcc3619198090ed49a286b7e2b6b3be5f04abb4d3539c: Status 404 returned error can't find the container with id 69a2a3308a45275d726bcc3619198090ed49a286b7e2b6b3be5f04abb4d3539c Sep 09 23:12:58.615788 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:58.615748 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tn22" event={"ID":"d5e895e5-ef52-4938-8fcf-e46232bc95a7","Type":"ContainerStarted","Data":"69a2a3308a45275d726bcc3619198090ed49a286b7e2b6b3be5f04abb4d3539c"} Sep 09 23:12:59.621129 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:59.621079 2571 generic.go:356] "Generic (PLEG): container finished" podID="d5e895e5-ef52-4938-8fcf-e46232bc95a7" containerID="df370e06fa628127986b519f1c5454d654ad2b2b1aafaecd8418742d2c7d1ec3" exitCode=0 Sep 09 23:12:59.621531 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:12:59.621176 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tn22" event={"ID":"d5e895e5-ef52-4938-8fcf-e46232bc95a7","Type":"ContainerDied","Data":"df370e06fa628127986b519f1c5454d654ad2b2b1aafaecd8418742d2c7d1ec3"} Sep 09 23:13:00.626509 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:00.626468 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tn22" event={"ID":"d5e895e5-ef52-4938-8fcf-e46232bc95a7","Type":"ContainerStarted","Data":"8ad3476985cf93078c3963f7570afbc4d16e5f5a3cdc56c9c815723b47b7e6d2"} Sep 09 23:13:00.626904 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:00.626524 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7tn22" event={"ID":"d5e895e5-ef52-4938-8fcf-e46232bc95a7","Type":"ContainerStarted","Data":"134de4bf9eadc366d5bba172a4c5609736b03280d8358fad1ffc216511e17edf"} Sep 09 23:13:00.648508 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:00.648455 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7tn22" podStartSLOduration=3.639805118 podStartE2EDuration="4.64843968s" totalImagesPullingTime="945.834044ms" totalInitContainerRuntime="62.800518ms" isStatefulPod=true podCreationTimestamp="2026-09-09 23:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:13:00.647715369 +0000 UTC m=+292.595127091" watchObservedRunningTime="2026-09-09 23:13:00.64843968 +0000 UTC m=+292.595851402" Sep 09 23:13:01.546095 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:13:01.546048 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-xbcm2" podUID="8647901c-c88a-4922-a37b-ecff32c52f6f" Sep 09 23:13:02.189379 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:02.189340 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" Sep 09 23:13:02.189785 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:02.189701 2571 scope.go:117] "RemoveContainer" containerID="4e6e7c5633eb594960b3bda1be91b05da1f6c2675ecf3464954202a99666a2db" Sep 09 23:13:02.189867 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:13:02.189848 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-d7fbd5d84-d82hg_open-cluster-management-agent-addon(fee92304-2731-4bc2-804d-16b2319e131e)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" podUID="fee92304-2731-4bc2-804d-16b2319e131e" Sep 09 23:13:02.207077 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:02.207045 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:13:02.207411 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:02.207399 2571 scope.go:117] "RemoveContainer" containerID="3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173" Sep 09 23:13:02.207584 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:13:02.207565 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-cd56cb79b-cclc2_open-cluster-management-agent-addon(c8d5ed99-1e6d-485e-94be-d1bb5d22137f)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" podUID="c8d5ed99-1e6d-485e-94be-d1bb5d22137f" Sep 09 23:13:06.613326 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:06.613295 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hgdps" Sep 09 23:13:08.427571 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:08.427545 2571 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 23:13:12.060530 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:12.060494 2571 patch_prober.go:28] interesting pod/image-registry-7464dc4644-c7hnn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 23:13:12.063970 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:12.060551 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 23:13:12.544648 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:12.544611 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:13:13.544689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.544654 2571 scope.go:117] "RemoveContainer" containerID="3cb10da54f26c01d2cd09b9ead4544143678bf8ada9cfd10364b27e011f9d173" Sep 09 23:13:13.544689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.544686 2571 scope.go:117] "RemoveContainer" containerID="4e6e7c5633eb594960b3bda1be91b05da1f6c2675ecf3464954202a99666a2db" Sep 09 23:13:13.596632 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.596607 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:13:13.676005 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.675970 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" event={"ID":"c8d5ed99-1e6d-485e-94be-d1bb5d22137f","Type":"ContainerStarted","Data":"05d1f1bdb44325eba10887df3f1f587e1921ad570aecfb30ec0c551dfe1217ad"} Sep 09 23:13:13.676520 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.676358 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:13:13.678751 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.678681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d7fbd5d84-d82hg" event={"ID":"fee92304-2731-4bc2-804d-16b2319e131e","Type":"ContainerStarted","Data":"012362955dfaa646e2d0fac61967c78671fd754f029b6156643cd79d02a7c3c5"} Sep 09 23:13:13.678751 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:13.678742 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cd56cb79b-cclc2" Sep 09 23:13:14.132315 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:14.132277 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7464dc4644-c7hnn"] Sep 09 23:13:39.152936 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.152891 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" containerName="registry" containerID="cri-o://c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c" gracePeriod=30 Sep 09 23:13:39.401806 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.401777 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:13:39.560464 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560422 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgkjc\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-kube-api-access-sgkjc\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.560464 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560467 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-installation-pull-secrets\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.560703 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560497 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-ca-trust-extracted\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.560703 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560526 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-image-registry-private-configuration\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.560806 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560699 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-trusted-ca\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.560806 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560763 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.560963 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.560938 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-bound-sa-token\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.561044 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.561022 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-certificates\") pod \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\" (UID: \"f20e2dbe-2a4e-4252-ab29-2dc666c8155d\") " Sep 09 23:13:39.561111 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.561067 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:39.561253 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.561234 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-trusted-ca\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.561650 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.561578 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:13:39.563092 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.563063 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:39.563389 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.563364 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-kube-api-access-sgkjc" (OuterVolumeSpecName: "kube-api-access-sgkjc") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "kube-api-access-sgkjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:13:39.563474 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.563455 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:13:39.563514 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.563459 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:13:39.563720 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.563696 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:13:39.570935 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.570908 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f20e2dbe-2a4e-4252-ab29-2dc666c8155d" (UID: "f20e2dbe-2a4e-4252-ab29-2dc666c8155d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:13:39.661817 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661776 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-tls\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.661817 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661811 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-bound-sa-token\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.661817 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661821 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-registry-certificates\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.662048 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661831 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgkjc\" (UniqueName: \"kubernetes.io/projected/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-kube-api-access-sgkjc\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.662048 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661840 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-installation-pull-secrets\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.662048 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661849 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-ca-trust-extracted\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.662048 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.661858 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f20e2dbe-2a4e-4252-ab29-2dc666c8155d-image-registry-private-configuration\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:13:39.769914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.769879 2571 generic.go:356] "Generic (PLEG): container finished" podID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" containerID="c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c" exitCode=0 Sep 09 23:13:39.770081 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.769935 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" Sep 09 23:13:39.770081 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.769952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" event={"ID":"f20e2dbe-2a4e-4252-ab29-2dc666c8155d","Type":"ContainerDied","Data":"c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c"} Sep 09 23:13:39.770081 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.770013 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7464dc4644-c7hnn" event={"ID":"f20e2dbe-2a4e-4252-ab29-2dc666c8155d","Type":"ContainerDied","Data":"2522783863d9af7b4b1e2fccf066275e7cb8ddfb60f84b8b03f0a6d33f0ce826"} Sep 09 23:13:39.770081 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.770031 2571 scope.go:117] "RemoveContainer" containerID="c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c" Sep 09 23:13:39.777981 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.777959 2571 scope.go:117] "RemoveContainer" containerID="c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c" Sep 09 23:13:39.778289 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:13:39.778266 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c\": container with ID starting with c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c not found: ID does not exist" containerID="c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c" Sep 09 23:13:39.778361 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.778296 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c"} err="failed to get container status \"c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c\": rpc error: code = NotFound desc = could not find container \"c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c\": container with ID starting with c23ce12a402443d9414bb3aefef4f2a360bf4d84c659216c93a953622d42927c not found: ID does not exist" Sep 09 23:13:39.791386 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.791360 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7464dc4644-c7hnn"] Sep 09 23:13:39.796795 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:39.796768 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7464dc4644-c7hnn"] Sep 09 23:13:40.548717 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:13:40.548684 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" path="/var/lib/kubelet/pods/f20e2dbe-2a4e-4252-ab29-2dc666c8155d/volumes" Sep 09 23:14:22.471371 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:22.471310 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:14:22.473597 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:22.473576 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8647901c-c88a-4922-a37b-ecff32c52f6f-metrics-certs\") pod \"network-metrics-daemon-xbcm2\" (UID: \"8647901c-c88a-4922-a37b-ecff32c52f6f\") " pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:14:22.749922 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:22.749834 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hgrbx\"" Sep 09 23:14:22.755328 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:22.755305 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbcm2" Sep 09 23:14:23.029514 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:23.029487 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbcm2"] Sep 09 23:14:23.031769 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:14:23.031741 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8647901c_c88a_4922_a37b_ecff32c52f6f.slice/crio-f2094607bb82dc72e7aa86e2624d761c7d712616321105f68686dcb9b0369828 WatchSource:0}: Error finding container f2094607bb82dc72e7aa86e2624d761c7d712616321105f68686dcb9b0369828: Status 404 returned error can't find the container with id f2094607bb82dc72e7aa86e2624d761c7d712616321105f68686dcb9b0369828 Sep 09 23:14:23.033622 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:23.033607 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:14:23.913688 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:23.913647 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbcm2" event={"ID":"8647901c-c88a-4922-a37b-ecff32c52f6f","Type":"ContainerStarted","Data":"f2094607bb82dc72e7aa86e2624d761c7d712616321105f68686dcb9b0369828"} Sep 09 23:14:24.918430 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:24.918386 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbcm2" event={"ID":"8647901c-c88a-4922-a37b-ecff32c52f6f","Type":"ContainerStarted","Data":"c7482da6de5c9501bce858534a8365b5e1a02e293576dc2063a7ab453add8e98"} Sep 09 23:14:24.918807 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:24.918453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbcm2" event={"ID":"8647901c-c88a-4922-a37b-ecff32c52f6f","Type":"ContainerStarted","Data":"c37ee959c908fac88b8c02406dbe652dd2a9c585d1c281dec0bb6112cb232ea3"} Sep 09 23:14:24.937375 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:14:24.937322 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xbcm2" podStartSLOduration=375.896565737 podStartE2EDuration="6m16.937307251s" totalImagesPullingTime="1.040741514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:14:24.93686021 +0000 UTC m=+376.884271933" watchObservedRunningTime="2026-09-09 23:14:24.937307251 +0000 UTC m=+376.884718974" Sep 09 23:17:08.752754 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.752649 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-b99hl"] Sep 09 23:17:08.753221 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.752871 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" containerName="registry" Sep 09 23:17:08.753221 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.752882 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" containerName="registry" Sep 09 23:17:08.753221 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.752932 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20e2dbe-2a4e-4252-ab29-2dc666c8155d" containerName="registry" Sep 09 23:17:08.755508 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.755492 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:08.758149 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.758108 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 23:17:08.759516 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.759500 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tqdsb\"" Sep 09 23:17:08.759589 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.759527 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 23:17:08.763195 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.763174 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-b99hl"] Sep 09 23:17:08.800131 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.800068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1225d197-0764-406f-ae26-a28dd08f3ce9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-b99hl\" (UID: \"1225d197-0764-406f-ae26-a28dd08f3ce9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:08.800305 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.800155 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/1225d197-0764-406f-ae26-a28dd08f3ce9-kube-api-access-pr46f\") pod \"cert-manager-cainjector-559cdd8c87-b99hl\" (UID: \"1225d197-0764-406f-ae26-a28dd08f3ce9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:08.901472 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.901433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1225d197-0764-406f-ae26-a28dd08f3ce9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-b99hl\" (UID: \"1225d197-0764-406f-ae26-a28dd08f3ce9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:08.901593 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.901491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/1225d197-0764-406f-ae26-a28dd08f3ce9-kube-api-access-pr46f\") pod \"cert-manager-cainjector-559cdd8c87-b99hl\" (UID: \"1225d197-0764-406f-ae26-a28dd08f3ce9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:08.909430 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.909391 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1225d197-0764-406f-ae26-a28dd08f3ce9-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-b99hl\" (UID: \"1225d197-0764-406f-ae26-a28dd08f3ce9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:08.909686 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:08.909664 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr46f\" (UniqueName: \"kubernetes.io/projected/1225d197-0764-406f-ae26-a28dd08f3ce9-kube-api-access-pr46f\") pod \"cert-manager-cainjector-559cdd8c87-b99hl\" (UID: \"1225d197-0764-406f-ae26-a28dd08f3ce9\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:09.064484 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:09.064449 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" Sep 09 23:17:09.329954 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:09.329930 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-b99hl"] Sep 09 23:17:09.332293 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:17:09.332255 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1225d197_0764_406f_ae26_a28dd08f3ce9.slice/crio-b0e64fb828bd4c05a3fd647be20ffe5ee0da679478fe3595631f9a12afaf95a0 WatchSource:0}: Error finding container b0e64fb828bd4c05a3fd647be20ffe5ee0da679478fe3595631f9a12afaf95a0: Status 404 returned error can't find the container with id b0e64fb828bd4c05a3fd647be20ffe5ee0da679478fe3595631f9a12afaf95a0 Sep 09 23:17:09.451016 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:09.450980 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" event={"ID":"1225d197-0764-406f-ae26-a28dd08f3ce9","Type":"ContainerStarted","Data":"b0e64fb828bd4c05a3fd647be20ffe5ee0da679478fe3595631f9a12afaf95a0"} Sep 09 23:17:13.468917 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:13.468879 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" event={"ID":"1225d197-0764-406f-ae26-a28dd08f3ce9","Type":"ContainerStarted","Data":"603065c91be5077707cbf274a7b8eb149f5fa1688d32f7058a6c53f2aab530c3"} Sep 09 23:17:13.485626 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:13.485560 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-b99hl" podStartSLOduration=2.263663283 podStartE2EDuration="5.485545187s" totalImagesPullingTime="3.221881904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:13.484616304 +0000 UTC m=+545.432028024" watchObservedRunningTime="2026-09-09 23:17:13.485545187 +0000 UTC m=+545.432956909" Sep 09 23:17:38.308874 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.308836 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z"] Sep 09 23:17:38.311767 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.311750 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.314969 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.314936 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 23:17:38.315270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.315253 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 23:17:38.315355 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.315287 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n2m42\"" Sep 09 23:17:38.315888 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.315868 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 23:17:38.318555 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.318538 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 23:17:38.335565 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.335531 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z"] Sep 09 23:17:38.417286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.417247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4365b36-0d59-4e67-9c77-311cec6242fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.417286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.417285 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-849s6\" (UniqueName: \"kubernetes.io/projected/c4365b36-0d59-4e67-9c77-311cec6242fd-kube-api-access-849s6\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.417504 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.417312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4365b36-0d59-4e67-9c77-311cec6242fd-manifests\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.417504 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.417366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4365b36-0d59-4e67-9c77-311cec6242fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.417504 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.417383 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4365b36-0d59-4e67-9c77-311cec6242fd-tmp\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.518282 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.518239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4365b36-0d59-4e67-9c77-311cec6242fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.518282 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.518276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-849s6\" (UniqueName: \"kubernetes.io/projected/c4365b36-0d59-4e67-9c77-311cec6242fd-kube-api-access-849s6\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.518483 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.518300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4365b36-0d59-4e67-9c77-311cec6242fd-manifests\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.518483 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.518326 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4365b36-0d59-4e67-9c77-311cec6242fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.518483 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.518350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4365b36-0d59-4e67-9c77-311cec6242fd-tmp\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.518741 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.518716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4365b36-0d59-4e67-9c77-311cec6242fd-manifests\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.520787 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.520764 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4365b36-0d59-4e67-9c77-311cec6242fd-tmp\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.521011 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.520985 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4365b36-0d59-4e67-9c77-311cec6242fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.521051 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.520985 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4365b36-0d59-4e67-9c77-311cec6242fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.526733 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.526709 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-849s6\" (UniqueName: \"kubernetes.io/projected/c4365b36-0d59-4e67-9c77-311cec6242fd-kube-api-access-849s6\") pod \"opendatahub-operator-controller-manager-68fd95c47c-zss8z\" (UID: \"c4365b36-0d59-4e67-9c77-311cec6242fd\") " pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.622354 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.622246 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:38.897275 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:38.897169 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z"] Sep 09 23:17:38.900646 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:17:38.900616 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4365b36_0d59_4e67_9c77_311cec6242fd.slice/crio-c9053e6a96f04140ea01515cabc0f6854887855d8a6a2ac3d02b722c86c69966 WatchSource:0}: Error finding container c9053e6a96f04140ea01515cabc0f6854887855d8a6a2ac3d02b722c86c69966: Status 404 returned error can't find the container with id c9053e6a96f04140ea01515cabc0f6854887855d8a6a2ac3d02b722c86c69966 Sep 09 23:17:39.557456 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:39.557417 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" event={"ID":"c4365b36-0d59-4e67-9c77-311cec6242fd","Type":"ContainerStarted","Data":"c9053e6a96f04140ea01515cabc0f6854887855d8a6a2ac3d02b722c86c69966"} Sep 09 23:17:42.570035 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:42.570000 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" event={"ID":"c4365b36-0d59-4e67-9c77-311cec6242fd","Type":"ContainerStarted","Data":"55ba9a6c842e9302235e738146bff56fb8614f6ba62ee0475586866f2eba1e36"} Sep 09 23:17:43.575271 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:43.575233 2571 generic.go:356] "Generic (PLEG): container finished" podID="c4365b36-0d59-4e67-9c77-311cec6242fd" containerID="55ba9a6c842e9302235e738146bff56fb8614f6ba62ee0475586866f2eba1e36" exitCode=0 Sep 09 23:17:43.575697 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:43.575308 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" event={"ID":"c4365b36-0d59-4e67-9c77-311cec6242fd","Type":"ContainerDied","Data":"55ba9a6c842e9302235e738146bff56fb8614f6ba62ee0475586866f2eba1e36"} Sep 09 23:17:44.580681 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:44.580647 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" event={"ID":"c4365b36-0d59-4e67-9c77-311cec6242fd","Type":"ContainerStarted","Data":"cea2691d36c2ecddb2ff7079e5054df4cc56500dd8d062f8073eefa4682c9ac5"} Sep 09 23:17:44.581082 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:44.580794 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:44.603988 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:44.603935 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" podStartSLOduration=2.342918478 podStartE2EDuration="6.603918707s" totalImagesPullingTime="3.200347078s" totalInitContainerRuntime="1.060653151s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:17:44.602058383 +0000 UTC m=+576.549470105" watchObservedRunningTime="2026-09-09 23:17:44.603918707 +0000 UTC m=+576.551330430" Sep 09 23:17:55.586519 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:55.586483 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68fd95c47c-zss8z" Sep 09 23:17:57.621289 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.621257 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn"] Sep 09 23:17:57.629271 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.629246 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.631978 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.631960 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 23:17:57.632239 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.632222 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 23:17:57.632292 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.632254 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ll5zx\"" Sep 09 23:17:57.633452 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.633429 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 23:17:57.633545 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.633472 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 23:17:57.638590 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.638568 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn"] Sep 09 23:17:57.754292 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.754252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f35310-390a-48fd-b29f-9c1d08b3dd81-tmp\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.754292 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.754302 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f35310-390a-48fd-b29f-9c1d08b3dd81-tls-certs\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.754526 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.754343 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6dz\" (UniqueName: \"kubernetes.io/projected/07f35310-390a-48fd-b29f-9c1d08b3dd81-kube-api-access-4w6dz\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.855492 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.855440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f35310-390a-48fd-b29f-9c1d08b3dd81-tmp\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.855492 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.855489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f35310-390a-48fd-b29f-9c1d08b3dd81-tls-certs\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.855759 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.855518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6dz\" (UniqueName: \"kubernetes.io/projected/07f35310-390a-48fd-b29f-9c1d08b3dd81-kube-api-access-4w6dz\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.855759 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:17:57.855604 2571 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 09 23:17:57.855759 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:17:57.855684 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07f35310-390a-48fd-b29f-9c1d08b3dd81-tls-certs podName:07f35310-390a-48fd-b29f-9c1d08b3dd81 nodeName:}" failed. No retries permitted until 2026-09-09 23:17:58.355668391 +0000 UTC m=+590.303080092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/07f35310-390a-48fd-b29f-9c1d08b3dd81-tls-certs") pod "kube-auth-proxy-576f68f9bc-wsjmn" (UID: "07f35310-390a-48fd-b29f-9c1d08b3dd81") : secret "kube-auth-proxy-tls" not found Sep 09 23:17:57.857778 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.857756 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f35310-390a-48fd-b29f-9c1d08b3dd81-tmp\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:57.877638 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:57.877572 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6dz\" (UniqueName: \"kubernetes.io/projected/07f35310-390a-48fd-b29f-9c1d08b3dd81-kube-api-access-4w6dz\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:58.359733 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:58.359694 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f35310-390a-48fd-b29f-9c1d08b3dd81-tls-certs\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:58.362179 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:58.362157 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f35310-390a-48fd-b29f-9c1d08b3dd81-tls-certs\") pod \"kube-auth-proxy-576f68f9bc-wsjmn\" (UID: \"07f35310-390a-48fd-b29f-9c1d08b3dd81\") " pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:58.539140 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:58.539083 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" Sep 09 23:17:58.810154 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:58.810044 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn"] Sep 09 23:17:58.812801 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:17:58.812776 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f35310_390a_48fd_b29f_9c1d08b3dd81.slice/crio-ec98a92668fbd251864346e0e0542e055b5a5439d6626c9990f33373839a192a WatchSource:0}: Error finding container ec98a92668fbd251864346e0e0542e055b5a5439d6626c9990f33373839a192a: Status 404 returned error can't find the container with id ec98a92668fbd251864346e0e0542e055b5a5439d6626c9990f33373839a192a Sep 09 23:17:59.637448 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:17:59.637319 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" event={"ID":"07f35310-390a-48fd-b29f-9c1d08b3dd81","Type":"ContainerStarted","Data":"ec98a92668fbd251864346e0e0542e055b5a5439d6626c9990f33373839a192a"} Sep 09 23:18:01.646710 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:01.646673 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" event={"ID":"07f35310-390a-48fd-b29f-9c1d08b3dd81","Type":"ContainerStarted","Data":"8892fd0451807818013e7a1f066e672296c13804a951bf7a47802bc19aa0bf96"} Sep 09 23:18:01.664880 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:01.664824 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-576f68f9bc-wsjmn" podStartSLOduration=2.007848862 podStartE2EDuration="4.664808612s" totalImagesPullingTime="2.65695975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:01.664580263 +0000 UTC m=+593.611991988" watchObservedRunningTime="2026-09-09 23:18:01.664808612 +0000 UTC m=+593.612220365" Sep 09 23:18:04.768445 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.768413 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6"] Sep 09 23:18:04.771845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.771817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.774852 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.774816 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 23:18:04.774978 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.774848 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gdgnv\"" Sep 09 23:18:04.784597 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.784566 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6"] Sep 09 23:18:04.910609 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910609 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910612 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910632 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xbs\" (UniqueName: \"kubernetes.io/projected/39309e3d-3900-4ecd-9bd9-dfab92e05360-kube-api-access-d8xbs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910786 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.910845 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39309e3d-3900-4ecd-9bd9-dfab92e05360-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:04.911087 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:04.910934 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.011964 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.011916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.011964 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.011960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.011987 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39309e3d-3900-4ecd-9bd9-dfab92e05360-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012046 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012100 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012158 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012270 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xbs\" (UniqueName: \"kubernetes.io/projected/39309e3d-3900-4ecd-9bd9-dfab92e05360-kube-api-access-d8xbs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012583 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012450 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012583 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012514 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012583 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012563 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012732 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012702 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.012867 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.012821 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39309e3d-3900-4ecd-9bd9-dfab92e05360-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.014607 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.014584 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.014793 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.014769 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.020617 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.020546 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39309e3d-3900-4ecd-9bd9-dfab92e05360-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.020735 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.020689 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xbs\" (UniqueName: \"kubernetes.io/projected/39309e3d-3900-4ecd-9bd9-dfab92e05360-kube-api-access-d8xbs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6\" (UID: \"39309e3d-3900-4ecd-9bd9-dfab92e05360\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.082570 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.082528 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:05.357610 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.357581 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6"] Sep 09 23:18:05.360062 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:18:05.360038 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39309e3d_3900_4ecd_9bd9_dfab92e05360.slice/crio-5988dc6ea4d6f3f69d5fa4dbe5cadba47e51dd9d3cd817019fd00dcf58212408 WatchSource:0}: Error finding container 5988dc6ea4d6f3f69d5fa4dbe5cadba47e51dd9d3cd817019fd00dcf58212408: Status 404 returned error can't find the container with id 5988dc6ea4d6f3f69d5fa4dbe5cadba47e51dd9d3cd817019fd00dcf58212408 Sep 09 23:18:05.662038 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:05.661952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" event={"ID":"39309e3d-3900-4ecd-9bd9-dfab92e05360","Type":"ContainerStarted","Data":"5988dc6ea4d6f3f69d5fa4dbe5cadba47e51dd9d3cd817019fd00dcf58212408"} Sep 09 23:18:08.201517 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.201479 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kcs88"] Sep 09 23:18:08.204969 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.204946 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:08.207888 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.207871 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 09 23:18:08.207961 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.207885 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wmh9k\"" Sep 09 23:18:08.208985 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.208950 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:18:08.209064 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.209021 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:18:08.209108 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.209062 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:18:08.217237 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.217204 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kcs88"] Sep 09 23:18:08.336057 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.336024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpr9\" (UniqueName: \"kubernetes.io/projected/2b53de38-ad45-40a1-956d-4bfedee7339c-kube-api-access-5lpr9\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:08.336237 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.336068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b53de38-ad45-40a1-956d-4bfedee7339c-cert\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:08.436472 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.436433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpr9\" (UniqueName: \"kubernetes.io/projected/2b53de38-ad45-40a1-956d-4bfedee7339c-kube-api-access-5lpr9\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:08.436657 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.436497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b53de38-ad45-40a1-956d-4bfedee7339c-cert\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:08.439311 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.439279 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 09 23:18:08.445795 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.445762 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpr9\" (UniqueName: \"kubernetes.io/projected/2b53de38-ad45-40a1-956d-4bfedee7339c-kube-api-access-5lpr9\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:08.446748 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:18:08.446727 2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 09 23:18:08.446883 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:18:08.446798 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b53de38-ad45-40a1-956d-4bfedee7339c-cert podName:2b53de38-ad45-40a1-956d-4bfedee7339c nodeName:}" failed. No retries permitted until 2026-09-09 23:18:08.946778006 +0000 UTC m=+600.894189723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2b53de38-ad45-40a1-956d-4bfedee7339c-cert") pod "odh-model-controller-7998c67dc4-kcs88" (UID: "2b53de38-ad45-40a1-956d-4bfedee7339c") : secret "odh-model-controller-webhook-cert" not found Sep 09 23:18:08.676572 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.676528 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" event={"ID":"39309e3d-3900-4ecd-9bd9-dfab92e05360","Type":"ContainerStarted","Data":"f1740a53db651b645f888f8a09cd072333084e8e1c9ad5553db70ad6544be107"} Sep 09 23:18:08.697805 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.697738 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" podStartSLOduration=1.851386041 podStartE2EDuration="4.697711267s" totalImagesPullingTime="2.846325226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:08.695858637 +0000 UTC m=+600.643270387" watchObservedRunningTime="2026-09-09 23:18:08.697711267 +0000 UTC m=+600.645122992" Sep 09 23:18:08.727000 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.726966 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n2htd"] Sep 09 23:18:08.731626 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.731604 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:08.735971 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.735952 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 09 23:18:08.746596 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.746561 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n2htd"] Sep 09 23:18:08.841081 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.841038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwssd\" (UniqueName: \"kubernetes.io/projected/e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d-kube-api-access-nwssd\") pod \"model-serving-api-54d89bd5fc-n2htd\" (UID: \"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:08.841294 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.841270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d-tls-certs\") pod \"model-serving-api-54d89bd5fc-n2htd\" (UID: \"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:08.942320 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.942233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwssd\" (UniqueName: \"kubernetes.io/projected/e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d-kube-api-access-nwssd\") pod \"model-serving-api-54d89bd5fc-n2htd\" (UID: \"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:08.942320 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.942281 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d-tls-certs\") pod \"model-serving-api-54d89bd5fc-n2htd\" (UID: \"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:08.944967 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.944939 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d-tls-certs\") pod \"model-serving-api-54d89bd5fc-n2htd\" (UID: \"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:08.950618 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:08.950586 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwssd\" (UniqueName: \"kubernetes.io/projected/e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d-kube-api-access-nwssd\") pod \"model-serving-api-54d89bd5fc-n2htd\" (UID: \"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d\") " pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:09.041795 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.041760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:09.043050 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.043019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b53de38-ad45-40a1-956d-4bfedee7339c-cert\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:09.045494 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.045467 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b53de38-ad45-40a1-956d-4bfedee7339c-cert\") pod \"odh-model-controller-7998c67dc4-kcs88\" (UID: \"2b53de38-ad45-40a1-956d-4bfedee7339c\") " pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:09.083535 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.083496 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:09.090210 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.089717 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:09.119934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.119901 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wmh9k\"" Sep 09 23:18:09.127515 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.127481 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:09.311182 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.311071 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-n2htd"] Sep 09 23:18:09.313333 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:18:09.313292 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9174d58_d8ff_455a_b0a1_e1df8cbd8f0d.slice/crio-c17c0deca8d2dd04b767d054360a3b846cd0ed05b087d493ba270225f185b4f3 WatchSource:0}: Error finding container c17c0deca8d2dd04b767d054360a3b846cd0ed05b087d493ba270225f185b4f3: Status 404 returned error can't find the container with id c17c0deca8d2dd04b767d054360a3b846cd0ed05b087d493ba270225f185b4f3 Sep 09 23:18:09.398277 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.398253 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kcs88"] Sep 09 23:18:09.401527 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:18:09.401493 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b53de38_ad45_40a1_956d_4bfedee7339c.slice/crio-2ac0614677862414ebc162a2355485f485410870a2f61298a2333133b588ae99 WatchSource:0}: Error finding container 2ac0614677862414ebc162a2355485f485410870a2f61298a2333133b588ae99: Status 404 returned error can't find the container with id 2ac0614677862414ebc162a2355485f485410870a2f61298a2333133b588ae99 Sep 09 23:18:09.681050 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.680947 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" event={"ID":"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d","Type":"ContainerStarted","Data":"c17c0deca8d2dd04b767d054360a3b846cd0ed05b087d493ba270225f185b4f3"} Sep 09 23:18:09.682083 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.682056 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" event={"ID":"2b53de38-ad45-40a1-956d-4bfedee7339c","Type":"ContainerStarted","Data":"2ac0614677862414ebc162a2355485f485410870a2f61298a2333133b588ae99"} Sep 09 23:18:09.682400 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.682373 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:09.683411 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:09.683393 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6" Sep 09 23:18:12.696749 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:12.696712 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" event={"ID":"2b53de38-ad45-40a1-956d-4bfedee7339c","Type":"ContainerStarted","Data":"6fd1313b724740d7ecca073d2ec818ceba04c63fade5e10196440f0c4963dbb7"} Sep 09 23:18:12.697269 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:12.696847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:12.698250 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:12.698229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" event={"ID":"e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d","Type":"ContainerStarted","Data":"f09de0a607231e5541dddf70731e08f338482b2b0dbbc834a54c908aa4e28b63"} Sep 09 23:18:12.698354 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:12.698340 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:18:12.718029 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:12.717966 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" podStartSLOduration=2.043568802 podStartE2EDuration="4.717946621s" totalImagesPullingTime="2.674377819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:12.716317978 +0000 UTC m=+604.663729703" watchObservedRunningTime="2026-09-09 23:18:12.717946621 +0000 UTC m=+604.665358344" Sep 09 23:18:12.734202 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:12.734148 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" podStartSLOduration=3.233816296 podStartE2EDuration="4.73410624s" totalImagesPullingTime="1.500289944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:12.733712929 +0000 UTC m=+604.681124652" watchObservedRunningTime="2026-09-09 23:18:12.73410624 +0000 UTC m=+604.681517963" Sep 09 23:18:14.999659 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:14.999624 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c"] Sep 09 23:18:15.005613 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.005586 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.008456 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.008425 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 23:18:15.008456 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.008448 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 23:18:15.008653 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.008426 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 23:18:15.008653 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.008436 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ww24w\"" Sep 09 23:18:15.020104 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.020074 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c"] Sep 09 23:18:15.095161 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.095161 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.095396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095197 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.095396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095230 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.095396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095249 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxn6j\" (UniqueName: \"kubernetes.io/projected/8d4da4b1-7550-42cd-a17e-e1390c57beba-kube-api-access-hxn6j\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.095396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095273 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4da4b1-7550-42cd-a17e-e1390c57beba-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.095396 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.095294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195746 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4da4b1-7550-42cd-a17e-e1390c57beba-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195812 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.195934 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.195920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxn6j\" (UniqueName: \"kubernetes.io/projected/8d4da4b1-7550-42cd-a17e-e1390c57beba-kube-api-access-hxn6j\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.196747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.196719 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.198187 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.198161 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4da4b1-7550-42cd-a17e-e1390c57beba-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.198342 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.198320 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.198527 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.198513 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.198633 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.198611 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.211634 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.211597 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxn6j\" (UniqueName: \"kubernetes.io/projected/8d4da4b1-7550-42cd-a17e-e1390c57beba-kube-api-access-hxn6j\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.213348 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.213324 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4da4b1-7550-42cd-a17e-e1390c57beba-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fj64c\" (UID: \"8d4da4b1-7550-42cd-a17e-e1390c57beba\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.315747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.315704 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:15.638273 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.638186 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c"] Sep 09 23:18:15.643558 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:18:15.643525 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4da4b1_7550_42cd_a17e_e1390c57beba.slice/crio-915e497e5484c2830a9d6d8c659abb40e1b31e4ea39e9486e2408582c79f719c WatchSource:0}: Error finding container 915e497e5484c2830a9d6d8c659abb40e1b31e4ea39e9486e2408582c79f719c: Status 404 returned error can't find the container with id 915e497e5484c2830a9d6d8c659abb40e1b31e4ea39e9486e2408582c79f719c Sep 09 23:18:15.711036 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:15.710994 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" event={"ID":"8d4da4b1-7550-42cd-a17e-e1390c57beba","Type":"ContainerStarted","Data":"915e497e5484c2830a9d6d8c659abb40e1b31e4ea39e9486e2408582c79f719c"} Sep 09 23:18:18.072713 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:18.072674 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:18:18.073013 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:18.072746 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:18:18.725056 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:18.725019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" event={"ID":"8d4da4b1-7550-42cd-a17e-e1390c57beba","Type":"ContainerStarted","Data":"d9c45fad7aeceff0d2927f2767d36e47294995743f9c8b65c4124baf8a78cb14"} Sep 09 23:18:18.725241 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:18.725147 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:18.758411 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:18.758349 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" podStartSLOduration=2.33164492 podStartE2EDuration="4.758333838s" totalImagesPullingTime="2.426688918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:18:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:18:18.754711455 +0000 UTC m=+610.702123191" watchObservedRunningTime="2026-09-09 23:18:18.758333838 +0000 UTC m=+610.705745561" Sep 09 23:18:19.730688 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:19.730659 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fj64c" Sep 09 23:18:23.704339 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:23.704306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kcs88" Sep 09 23:18:23.706550 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:18:23.706527 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-n2htd" Sep 09 23:19:33.046666 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.046629 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hp6cv"] Sep 09 23:19:33.049832 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.049812 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:33.052994 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.052970 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 23:19:33.053105 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.053003 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 23:19:33.053105 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.053038 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bnz58\"" Sep 09 23:19:33.059991 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.059960 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hp6cv"] Sep 09 23:19:33.111345 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.111297 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrkj\" (UniqueName: \"kubernetes.io/projected/d39c290f-5941-4870-abbc-e4ea35964671-kube-api-access-2nrkj\") pod \"authorino-operator-76b478bd78-hp6cv\" (UID: \"d39c290f-5941-4870-abbc-e4ea35964671\") " pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:33.212206 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.212167 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrkj\" (UniqueName: \"kubernetes.io/projected/d39c290f-5941-4870-abbc-e4ea35964671-kube-api-access-2nrkj\") pod \"authorino-operator-76b478bd78-hp6cv\" (UID: \"d39c290f-5941-4870-abbc-e4ea35964671\") " pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:33.224689 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.224654 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrkj\" (UniqueName: \"kubernetes.io/projected/d39c290f-5941-4870-abbc-e4ea35964671-kube-api-access-2nrkj\") pod \"authorino-operator-76b478bd78-hp6cv\" (UID: \"d39c290f-5941-4870-abbc-e4ea35964671\") " pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:33.361264 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.361164 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:33.633185 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.633161 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hp6cv"] Sep 09 23:19:33.637801 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:19:33.637767 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39c290f_5941_4870_abbc_e4ea35964671.slice/crio-5ee745bd3d16ade242ec95c11d18eafe2504334ff52753f1af4fb1e171bc05eb WatchSource:0}: Error finding container 5ee745bd3d16ade242ec95c11d18eafe2504334ff52753f1af4fb1e171bc05eb: Status 404 returned error can't find the container with id 5ee745bd3d16ade242ec95c11d18eafe2504334ff52753f1af4fb1e171bc05eb Sep 09 23:19:33.640963 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:33.640944 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:19:34.027785 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:34.027702 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" event={"ID":"d39c290f-5941-4870-abbc-e4ea35964671","Type":"ContainerStarted","Data":"5ee745bd3d16ade242ec95c11d18eafe2504334ff52753f1af4fb1e171bc05eb"} Sep 09 23:19:37.044873 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:37.044837 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" event={"ID":"d39c290f-5941-4870-abbc-e4ea35964671","Type":"ContainerStarted","Data":"b81c129832cdb8b8c26c338ee3b638c14579d504eb745bb87e3b415be5774c70"} Sep 09 23:19:37.045316 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:37.044962 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:37.064273 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:37.064219 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" podStartSLOduration=1.552239733 podStartE2EDuration="4.064200733s" totalImagesPullingTime="2.511961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:19:37.062893243 +0000 UTC m=+689.010304970" watchObservedRunningTime="2026-09-09 23:19:37.064200733 +0000 UTC m=+689.011612456" Sep 09 23:19:47.530404 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.530369 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs"] Sep 09 23:19:47.533726 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.533703 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.537173 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.537148 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hp8pg\"" Sep 09 23:19:47.548613 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.548584 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs"] Sep 09 23:19:47.621509 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.621466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776d7a33-6c7f-4b9e-90b0-93371549eaaf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.621509 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.621518 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrvg\" (UniqueName: \"kubernetes.io/projected/776d7a33-6c7f-4b9e-90b0-93371549eaaf-kube-api-access-zcrvg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.721910 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.721869 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrvg\" (UniqueName: \"kubernetes.io/projected/776d7a33-6c7f-4b9e-90b0-93371549eaaf-kube-api-access-zcrvg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.722058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.721938 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776d7a33-6c7f-4b9e-90b0-93371549eaaf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.722287 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.722271 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776d7a33-6c7f-4b9e-90b0-93371549eaaf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.730081 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.730059 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrvg\" (UniqueName: \"kubernetes.io/projected/776d7a33-6c7f-4b9e-90b0-93371549eaaf-kube-api-access-zcrvg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.843093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.843055 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:47.881548 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.881511 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs"] Sep 09 23:19:47.894865 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.894832 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs"] Sep 09 23:19:47.911236 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.910771 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk"] Sep 09 23:19:47.915864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.915839 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:47.939328 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:47.939291 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk"] Sep 09 23:19:48.024714 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.024680 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6996628-da16-4410-aab8-7c5bd26eb00a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-2wngk\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.024871 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.024727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nw24\" (UniqueName: \"kubernetes.io/projected/b6996628-da16-4410-aab8-7c5bd26eb00a-kube-api-access-2nw24\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-2wngk\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.044015 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:19:48.043977 2571 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 09 23:19:48.044015 ip-10-0-140-142 kubenswrapper[2571]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-vx2qs_kuadrant-system_776d7a33-6c7f-4b9e-90b0-93371549eaaf_0(2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-vx2qs to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1" Netns:"/var/run/netns/03da3081-3c6a-4f14-bdda-5a0a446ffe8a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-vx2qs;K8S_POD_INFRA_CONTAINER_ID=2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1;K8S_POD_UID=776d7a33-6c7f-4b9e-90b0-93371549eaaf" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs/776d7a33-6c7f-4b9e-90b0-93371549eaaf:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs 2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:16 [10.132.0.22/23] Sep 09 23:19:48.044015 ip-10-0-140-142 kubenswrapper[2571]: ' Sep 09 23:19:48.044015 ip-10-0-140-142 kubenswrapper[2571]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 09 23:19:48.044015 ip-10-0-140-142 kubenswrapper[2571]: > Sep 09 23:19:48.044237 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:19:48.044044 2571 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 09 23:19:48.044237 ip-10-0-140-142 kubenswrapper[2571]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-vx2qs_kuadrant-system_776d7a33-6c7f-4b9e-90b0-93371549eaaf_0(2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-vx2qs to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1" Netns:"/var/run/netns/03da3081-3c6a-4f14-bdda-5a0a446ffe8a" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-vx2qs;K8S_POD_INFRA_CONTAINER_ID=2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1;K8S_POD_UID=776d7a33-6c7f-4b9e-90b0-93371549eaaf" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs/776d7a33-6c7f-4b9e-90b0-93371549eaaf:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs 2be6f26309ee93e766320dbe7978dd577987491763a5812f0bcb871617550ef1 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:16 [10.132.0.22/23] Sep 09 23:19:48.044237 ip-10-0-140-142 kubenswrapper[2571]: ' Sep 09 23:19:48.044237 ip-10-0-140-142 kubenswrapper[2571]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 09 23:19:48.044237 ip-10-0-140-142 kubenswrapper[2571]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:48.052076 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.052051 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-hp6cv" Sep 09 23:19:48.089245 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.089214 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:48.091809 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:19:48.091776 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" is forbidden: User \"system:node:ip-10-0-140-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-142.ec2.internal' and this object" podUID="776d7a33-6c7f-4b9e-90b0-93371549eaaf" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:48.097470 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.097400 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:48.099890 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:19:48.099860 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" is forbidden: User \"system:node:ip-10-0-140-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-142.ec2.internal' and this object" podUID="776d7a33-6c7f-4b9e-90b0-93371549eaaf" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:48.126068 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.126028 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcrvg\" (UniqueName: \"kubernetes.io/projected/776d7a33-6c7f-4b9e-90b0-93371549eaaf-kube-api-access-zcrvg\") pod \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " Sep 09 23:19:48.126240 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.126127 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776d7a33-6c7f-4b9e-90b0-93371549eaaf-extensions-socket-volume\") pod \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\" (UID: \"776d7a33-6c7f-4b9e-90b0-93371549eaaf\") " Sep 09 23:19:48.126313 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.126295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6996628-da16-4410-aab8-7c5bd26eb00a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-2wngk\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.126349 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.126333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nw24\" (UniqueName: \"kubernetes.io/projected/b6996628-da16-4410-aab8-7c5bd26eb00a-kube-api-access-2nw24\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-2wngk\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.126392 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.126346 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d7a33-6c7f-4b9e-90b0-93371549eaaf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "776d7a33-6c7f-4b9e-90b0-93371549eaaf" (UID: "776d7a33-6c7f-4b9e-90b0-93371549eaaf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:19:48.126702 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.126677 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6996628-da16-4410-aab8-7c5bd26eb00a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-2wngk\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.128212 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.128191 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776d7a33-6c7f-4b9e-90b0-93371549eaaf-kube-api-access-zcrvg" (OuterVolumeSpecName: "kube-api-access-zcrvg") pod "776d7a33-6c7f-4b9e-90b0-93371549eaaf" (UID: "776d7a33-6c7f-4b9e-90b0-93371549eaaf"). InnerVolumeSpecName "kube-api-access-zcrvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:19:48.135256 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.135228 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nw24\" (UniqueName: \"kubernetes.io/projected/b6996628-da16-4410-aab8-7c5bd26eb00a-kube-api-access-2nw24\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-2wngk\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.227282 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.227234 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcrvg\" (UniqueName: \"kubernetes.io/projected/776d7a33-6c7f-4b9e-90b0-93371549eaaf-kube-api-access-zcrvg\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:19:48.227282 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.227271 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/776d7a33-6c7f-4b9e-90b0-93371549eaaf-extensions-socket-volume\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:19:48.228139 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.228090 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:48.503487 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.503461 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk"] Sep 09 23:19:48.506143 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:19:48.506107 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6996628_da16_4410_aab8_7c5bd26eb00a.slice/crio-4c2d0221772a1a0672a84a5131ab390787a8bd0690c17852c3854b784f496ab6 WatchSource:0}: Error finding container 4c2d0221772a1a0672a84a5131ab390787a8bd0690c17852c3854b784f496ab6: Status 404 returned error can't find the container with id 4c2d0221772a1a0672a84a5131ab390787a8bd0690c17852c3854b784f496ab6 Sep 09 23:19:48.549246 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:48.549209 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="776d7a33-6c7f-4b9e-90b0-93371549eaaf" path="/var/lib/kubelet/pods/776d7a33-6c7f-4b9e-90b0-93371549eaaf/volumes" Sep 09 23:19:48.550400 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:19:48.550370 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" is forbidden: User \"system:node:ip-10-0-140-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-142.ec2.internal' and this object" podUID="776d7a33-6c7f-4b9e-90b0-93371549eaaf" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:49.095085 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:49.095023 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" event={"ID":"b6996628-da16-4410-aab8-7c5bd26eb00a","Type":"ContainerStarted","Data":"4c2d0221772a1a0672a84a5131ab390787a8bd0690c17852c3854b784f496ab6"} Sep 09 23:19:49.095085 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:49.095042 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:49.099770 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:19:49.099734 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vx2qs\" is forbidden: User \"system:node:ip-10-0-140-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-142.ec2.internal' and this object" podUID="776d7a33-6c7f-4b9e-90b0-93371549eaaf" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vx2qs" Sep 09 23:19:54.120535 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:54.120500 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" event={"ID":"b6996628-da16-4410-aab8-7c5bd26eb00a","Type":"ContainerStarted","Data":"b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8"} Sep 09 23:19:54.120931 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:54.120623 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:19:54.145338 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:19:54.145284 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" podStartSLOduration=2.550168015 podStartE2EDuration="7.145268177s" totalImagesPullingTime="4.595100162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:19:54.143286636 +0000 UTC m=+706.090698358" watchObservedRunningTime="2026-09-09 23:19:54.145268177 +0000 UTC m=+706.092679900" Sep 09 23:20:05.126829 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:05.126741 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:20:17.917550 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:17.917514 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk"] Sep 09 23:20:17.918039 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:17.917734 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" podUID="b6996628-da16-4410-aab8-7c5bd26eb00a" containerName="manager" containerID="cri-o://b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8" gracePeriod=10 Sep 09 23:20:18.164667 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.164637 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:20:18.222755 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.222665 2571 generic.go:356] "Generic (PLEG): container finished" podID="b6996628-da16-4410-aab8-7c5bd26eb00a" containerID="b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8" exitCode=0 Sep 09 23:20:18.222755 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.222713 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" Sep 09 23:20:18.222976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.222760 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" event={"ID":"b6996628-da16-4410-aab8-7c5bd26eb00a","Type":"ContainerDied","Data":"b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8"} Sep 09 23:20:18.222976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.222827 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk" event={"ID":"b6996628-da16-4410-aab8-7c5bd26eb00a","Type":"ContainerDied","Data":"4c2d0221772a1a0672a84a5131ab390787a8bd0690c17852c3854b784f496ab6"} Sep 09 23:20:18.222976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.222858 2571 scope.go:117] "RemoveContainer" containerID="b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8" Sep 09 23:20:18.231084 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.231062 2571 scope.go:117] "RemoveContainer" containerID="b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8" Sep 09 23:20:18.231414 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:20:18.231392 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8\": container with ID starting with b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8 not found: ID does not exist" containerID="b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8" Sep 09 23:20:18.231479 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.231421 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8"} err="failed to get container status \"b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8\": rpc error: code = NotFound desc = could not find container \"b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8\": container with ID starting with b0a655a7ec2c7419b3b304bc1923606aa7a88bb955e98c5a0f9859d85665b1d8 not found: ID does not exist" Sep 09 23:20:18.270780 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.270745 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nw24\" (UniqueName: \"kubernetes.io/projected/b6996628-da16-4410-aab8-7c5bd26eb00a-kube-api-access-2nw24\") pod \"b6996628-da16-4410-aab8-7c5bd26eb00a\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " Sep 09 23:20:18.270909 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.270822 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6996628-da16-4410-aab8-7c5bd26eb00a-extensions-socket-volume\") pod \"b6996628-da16-4410-aab8-7c5bd26eb00a\" (UID: \"b6996628-da16-4410-aab8-7c5bd26eb00a\") " Sep 09 23:20:18.271254 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.271226 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6996628-da16-4410-aab8-7c5bd26eb00a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b6996628-da16-4410-aab8-7c5bd26eb00a" (UID: "b6996628-da16-4410-aab8-7c5bd26eb00a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 23:20:18.273027 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.273006 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6996628-da16-4410-aab8-7c5bd26eb00a-kube-api-access-2nw24" (OuterVolumeSpecName: "kube-api-access-2nw24") pod "b6996628-da16-4410-aab8-7c5bd26eb00a" (UID: "b6996628-da16-4410-aab8-7c5bd26eb00a"). InnerVolumeSpecName "kube-api-access-2nw24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:20:18.371658 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.371619 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6996628-da16-4410-aab8-7c5bd26eb00a-extensions-socket-volume\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:20:18.371658 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.371651 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nw24\" (UniqueName: \"kubernetes.io/projected/b6996628-da16-4410-aab8-7c5bd26eb00a-kube-api-access-2nw24\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:20:18.549745 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.549711 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk"] Sep 09 23:20:18.549745 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:18.549748 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-2wngk"] Sep 09 23:20:20.548960 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:20.548921 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6996628-da16-4410-aab8-7c5bd26eb00a" path="/var/lib/kubelet/pods/b6996628-da16-4410-aab8-7c5bd26eb00a/volumes" Sep 09 23:20:21.476052 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.476010 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc"] Sep 09 23:20:21.476401 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.476384 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6996628-da16-4410-aab8-7c5bd26eb00a" containerName="manager" Sep 09 23:20:21.476482 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.476405 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6996628-da16-4410-aab8-7c5bd26eb00a" containerName="manager" Sep 09 23:20:21.476482 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.476478 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6996628-da16-4410-aab8-7c5bd26eb00a" containerName="manager" Sep 09 23:20:21.481007 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.480983 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.483847 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.483828 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cdpjt\"" Sep 09 23:20:21.491942 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.491909 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc"] Sep 09 23:20:21.602375 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602375 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46r59\" (UniqueName: \"kubernetes.io/projected/751437d2-bef6-4c18-b471-6514a7dcb80b-kube-api-access-46r59\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602444 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751437d2-bef6-4c18-b471-6514a7dcb80b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602636 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.602801 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.602672 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703682 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703637 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703695 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751437d2-bef6-4c18-b471-6514a7dcb80b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703715 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703735 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703755 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46r59\" (UniqueName: \"kubernetes.io/projected/751437d2-bef6-4c18-b471-6514a7dcb80b-kube-api-access-46r59\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.703868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.703848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.704286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.704181 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.704286 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.704251 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.704520 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.704497 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.704621 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.704557 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751437d2-bef6-4c18-b471-6514a7dcb80b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.704702 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.704687 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.706030 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.705986 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.707019 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.706988 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.713058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.713026 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751437d2-bef6-4c18-b471-6514a7dcb80b-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.713383 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.713365 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46r59\" (UniqueName: \"kubernetes.io/projected/751437d2-bef6-4c18-b471-6514a7dcb80b-kube-api-access-46r59\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nhpdc\" (UID: \"751437d2-bef6-4c18-b471-6514a7dcb80b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:21.791738 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:21.791699 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:22.067175 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.067148 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc"] Sep 09 23:20:22.069170 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:20:22.069139 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751437d2_bef6_4c18_b471_6514a7dcb80b.slice/crio-14e6414b3034d3cb311ee5c2e316eeab88eac20937b13f9040b7cc21ffa55d52 WatchSource:0}: Error finding container 14e6414b3034d3cb311ee5c2e316eeab88eac20937b13f9040b7cc21ffa55d52: Status 404 returned error can't find the container with id 14e6414b3034d3cb311ee5c2e316eeab88eac20937b13f9040b7cc21ffa55d52 Sep 09 23:20:22.071345 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.071314 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:20:22.071412 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.071382 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:20:22.071447 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.071411 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:20:22.244293 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.244252 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" event={"ID":"751437d2-bef6-4c18-b471-6514a7dcb80b","Type":"ContainerStarted","Data":"0c28ced0aff11746940f3f1f520fb36f4186e7211575fdb1c97f01ae8f503234"} Sep 09 23:20:22.244472 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.244313 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" event={"ID":"751437d2-bef6-4c18-b471-6514a7dcb80b","Type":"ContainerStarted","Data":"14e6414b3034d3cb311ee5c2e316eeab88eac20937b13f9040b7cc21ffa55d52"} Sep 09 23:20:22.266646 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.266581 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" podStartSLOduration=1.266556685 podStartE2EDuration="1.266556685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:20:22.26447174 +0000 UTC m=+734.211883463" watchObservedRunningTime="2026-09-09 23:20:22.266556685 +0000 UTC m=+734.213968408" Sep 09 23:20:22.792238 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:22.792193 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:23.797046 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:23.797012 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:24.253211 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:24.253132 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:24.254280 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:24.254257 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:20:36.319589 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.319553 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:36.322849 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.322827 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.325603 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.325573 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 23:20:36.325736 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.325650 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gqj2n\"" Sep 09 23:20:36.330100 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.330070 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:36.415194 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.415162 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:36.421874 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.421844 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-config-file\") pod \"limitador-limitador-8588749977-l2jlm\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.421992 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.421920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqq44\" (UniqueName: \"kubernetes.io/projected/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-kube-api-access-jqq44\") pod \"limitador-limitador-8588749977-l2jlm\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.522924 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.522882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-config-file\") pod \"limitador-limitador-8588749977-l2jlm\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.523152 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.523098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqq44\" (UniqueName: \"kubernetes.io/projected/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-kube-api-access-jqq44\") pod \"limitador-limitador-8588749977-l2jlm\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.523812 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.523788 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-config-file\") pod \"limitador-limitador-8588749977-l2jlm\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.531725 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.531664 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqq44\" (UniqueName: \"kubernetes.io/projected/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-kube-api-access-jqq44\") pod \"limitador-limitador-8588749977-l2jlm\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.633799 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.633723 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:36.907245 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:36.907215 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:36.909277 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:20:36.909244 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b70b8d_6bbe_4734_adb2_5a7305b63bbf.slice/crio-ef24f0b66874c35bc795b473ec6f0263c0ef2a3e9931ccf60fa830a0c690b661 WatchSource:0}: Error finding container ef24f0b66874c35bc795b473ec6f0263c0ef2a3e9931ccf60fa830a0c690b661: Status 404 returned error can't find the container with id ef24f0b66874c35bc795b473ec6f0263c0ef2a3e9931ccf60fa830a0c690b661 Sep 09 23:20:37.312834 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:37.312791 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" event={"ID":"08b70b8d-6bbe-4734-adb2-5a7305b63bbf","Type":"ContainerStarted","Data":"ef24f0b66874c35bc795b473ec6f0263c0ef2a3e9931ccf60fa830a0c690b661"} Sep 09 23:20:38.318614 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:38.318523 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" event={"ID":"08b70b8d-6bbe-4734-adb2-5a7305b63bbf","Type":"ContainerStarted","Data":"b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99"} Sep 09 23:20:38.318614 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:38.318588 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:38.339088 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:38.339030 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" podStartSLOduration=1.189682766 podStartE2EDuration="2.339014907s" totalImagesPullingTime="1.149332141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:20:38.33776944 +0000 UTC m=+750.285181163" watchObservedRunningTime="2026-09-09 23:20:38.339014907 +0000 UTC m=+750.286426653" Sep 09 23:20:49.323610 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:49.323573 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:51.488868 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:51.488831 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:51.489295 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:51.489063 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" podUID="08b70b8d-6bbe-4734-adb2-5a7305b63bbf" containerName="limitador" containerID="cri-o://b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99" gracePeriod=30 Sep 09 23:20:52.037552 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.037519 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:52.144640 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.144554 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqq44\" (UniqueName: \"kubernetes.io/projected/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-kube-api-access-jqq44\") pod \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " Sep 09 23:20:52.144640 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.144637 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-config-file\") pod \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\" (UID: \"08b70b8d-6bbe-4734-adb2-5a7305b63bbf\") " Sep 09 23:20:52.145022 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.144989 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-config-file" (OuterVolumeSpecName: "config-file") pod "08b70b8d-6bbe-4734-adb2-5a7305b63bbf" (UID: "08b70b8d-6bbe-4734-adb2-5a7305b63bbf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 23:20:52.146556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.146535 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-kube-api-access-jqq44" (OuterVolumeSpecName: "kube-api-access-jqq44") pod "08b70b8d-6bbe-4734-adb2-5a7305b63bbf" (UID: "08b70b8d-6bbe-4734-adb2-5a7305b63bbf"). InnerVolumeSpecName "kube-api-access-jqq44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:20:52.245363 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.245322 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqq44\" (UniqueName: \"kubernetes.io/projected/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-kube-api-access-jqq44\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:20:52.245363 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.245355 2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/08b70b8d-6bbe-4734-adb2-5a7305b63bbf-config-file\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:20:52.378977 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.378939 2571 generic.go:356] "Generic (PLEG): container finished" podID="08b70b8d-6bbe-4734-adb2-5a7305b63bbf" containerID="b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99" exitCode=0 Sep 09 23:20:52.379183 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.378984 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" Sep 09 23:20:52.379183 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.379020 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" event={"ID":"08b70b8d-6bbe-4734-adb2-5a7305b63bbf","Type":"ContainerDied","Data":"b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99"} Sep 09 23:20:52.379183 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.379069 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-l2jlm" event={"ID":"08b70b8d-6bbe-4734-adb2-5a7305b63bbf","Type":"ContainerDied","Data":"ef24f0b66874c35bc795b473ec6f0263c0ef2a3e9931ccf60fa830a0c690b661"} Sep 09 23:20:52.379183 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.379097 2571 scope.go:117] "RemoveContainer" containerID="b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99" Sep 09 23:20:52.387453 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.387433 2571 scope.go:117] "RemoveContainer" containerID="b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99" Sep 09 23:20:52.387760 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:20:52.387741 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99\": container with ID starting with b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99 not found: ID does not exist" containerID="b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99" Sep 09 23:20:52.387811 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.387770 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99"} err="failed to get container status \"b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99\": rpc error: code = NotFound desc = could not find container \"b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99\": container with ID starting with b37f5045875f2d9ae23b039166c25cfa78d15a8f91665fd9cfc474cd3b351b99 not found: ID does not exist" Sep 09 23:20:52.399813 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.399744 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:52.403760 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.403735 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-l2jlm"] Sep 09 23:20:52.549354 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:52.549323 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b70b8d-6bbe-4734-adb2-5a7305b63bbf" path="/var/lib/kubelet/pods/08b70b8d-6bbe-4734-adb2-5a7305b63bbf/volumes" Sep 09 23:20:55.174646 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.174613 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gncql"] Sep 09 23:20:55.175058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.174897 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b70b8d-6bbe-4734-adb2-5a7305b63bbf" containerName="limitador" Sep 09 23:20:55.175058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.174907 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b70b8d-6bbe-4734-adb2-5a7305b63bbf" containerName="limitador" Sep 09 23:20:55.175058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.174971 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b70b8d-6bbe-4734-adb2-5a7305b63bbf" containerName="limitador" Sep 09 23:20:55.179068 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.179046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.181971 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.181949 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 09 23:20:55.182066 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.181949 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vrgf8\"" Sep 09 23:20:55.183600 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.183576 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 23:20:55.184074 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.184048 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 23:20:55.185359 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.185338 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gncql"] Sep 09 23:20:55.269393 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.269355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6661d673-1114-4a2d-89d3-dcbc79d90b56-data\") pod \"postgres-69fb4594cf-gncql\" (UID: \"6661d673-1114-4a2d-89d3-dcbc79d90b56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.269576 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.269444 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6vb\" (UniqueName: \"kubernetes.io/projected/6661d673-1114-4a2d-89d3-dcbc79d90b56-kube-api-access-qt6vb\") pod \"postgres-69fb4594cf-gncql\" (UID: \"6661d673-1114-4a2d-89d3-dcbc79d90b56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.370818 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.370773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6661d673-1114-4a2d-89d3-dcbc79d90b56-data\") pod \"postgres-69fb4594cf-gncql\" (UID: \"6661d673-1114-4a2d-89d3-dcbc79d90b56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.370998 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.370844 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6vb\" (UniqueName: \"kubernetes.io/projected/6661d673-1114-4a2d-89d3-dcbc79d90b56-kube-api-access-qt6vb\") pod \"postgres-69fb4594cf-gncql\" (UID: \"6661d673-1114-4a2d-89d3-dcbc79d90b56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.371201 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.371179 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/6661d673-1114-4a2d-89d3-dcbc79d90b56-data\") pod \"postgres-69fb4594cf-gncql\" (UID: \"6661d673-1114-4a2d-89d3-dcbc79d90b56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.378929 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.378892 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6vb\" (UniqueName: \"kubernetes.io/projected/6661d673-1114-4a2d-89d3-dcbc79d90b56-kube-api-access-qt6vb\") pod \"postgres-69fb4594cf-gncql\" (UID: \"6661d673-1114-4a2d-89d3-dcbc79d90b56\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.490003 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.489894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:20:55.758164 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:55.758066 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gncql"] Sep 09 23:20:55.761466 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:20:55.761435 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6661d673_1114_4a2d_89d3_dcbc79d90b56.slice/crio-1dac219800de713fd85659a74c27a6b0d7607112acdfd00d78c87faee11f851d WatchSource:0}: Error finding container 1dac219800de713fd85659a74c27a6b0d7607112acdfd00d78c87faee11f851d: Status 404 returned error can't find the container with id 1dac219800de713fd85659a74c27a6b0d7607112acdfd00d78c87faee11f851d Sep 09 23:20:56.398685 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:20:56.398643 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" event={"ID":"6661d673-1114-4a2d-89d3-dcbc79d90b56","Type":"ContainerStarted","Data":"1dac219800de713fd85659a74c27a6b0d7607112acdfd00d78c87faee11f851d"} Sep 09 23:21:01.424877 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:01.424837 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" event={"ID":"6661d673-1114-4a2d-89d3-dcbc79d90b56","Type":"ContainerStarted","Data":"6f577e494449aab1cca1ac1c774abfc5a88929f46321c0cf7b5b9686a1a1be68"} Sep 09 23:21:01.425555 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:01.424937 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:21:01.444548 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:01.444472 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" podStartSLOduration=1.756513872 podStartE2EDuration="6.4444474s" totalImagesPullingTime="4.687933528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:20:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:21:01.440791145 +0000 UTC m=+773.388202869" watchObservedRunningTime="2026-09-09 23:21:01.4444474 +0000 UTC m=+773.391859124" Sep 09 23:21:07.457955 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:07.457922 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gncql" Sep 09 23:21:14.924375 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.924339 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c795cf5d5-m9j5f"] Sep 09 23:21:14.947390 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.947348 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c795cf5d5-m9j5f"] Sep 09 23:21:14.947546 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.947504 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:14.950633 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.950559 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 23:21:14.951058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.951018 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 23:21:14.955186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.951271 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qjgr7\"" Sep 09 23:21:14.955186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:14.952827 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 23:21:15.039371 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.039327 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.039554 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.039401 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.039554 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.039445 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp68g\" (UniqueName: \"kubernetes.io/projected/595df9aa-c1da-4462-b28e-f8a65a86df44-kube-api-access-tp68g\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.104450 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.104405 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-847796847b-rd9zd"] Sep 09 23:21:15.107924 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.107902 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.118021 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.117989 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-847796847b-rd9zd"] Sep 09 23:21:15.140776 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.140733 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.140776 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.140777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp68g\" (UniqueName: \"kubernetes.io/projected/595df9aa-c1da-4462-b28e-f8a65a86df44-kube-api-access-tp68g\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.141025 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.140813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.141025 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:15.140919 2571 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 09 23:21:15.141025 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:15.140920 2571 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 09 23:21:15.141194 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:15.140996 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert podName:595df9aa-c1da-4462-b28e-f8a65a86df44 nodeName:}" failed. No retries permitted until 2026-09-09 23:21:15.640975709 +0000 UTC m=+787.588387410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert") pod "maas-controller-c795cf5d5-m9j5f" (UID: "595df9aa-c1da-4462-b28e-f8a65a86df44") : secret "maas-controller-webhook-cert" not found Sep 09 23:21:15.141194 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:15.141069 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs podName:595df9aa-c1da-4462-b28e-f8a65a86df44 nodeName:}" failed. No retries permitted until 2026-09-09 23:21:15.641048152 +0000 UTC m=+787.588459853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs") pod "maas-controller-c795cf5d5-m9j5f" (UID: "595df9aa-c1da-4462-b28e-f8a65a86df44") : secret "maas-controller-metrics-tls" not found Sep 09 23:21:15.151741 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.151710 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp68g\" (UniqueName: \"kubernetes.io/projected/595df9aa-c1da-4462-b28e-f8a65a86df44-kube-api-access-tp68g\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.241761 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.241672 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-metrics-certs\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.241761 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.241712 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbnb6\" (UniqueName: \"kubernetes.io/projected/8312264f-5e72-43fa-babf-d88e31edcccc-kube-api-access-gbnb6\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.241761 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.241741 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-webhook-cert\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.342244 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.342199 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-metrics-certs\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.342244 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.342244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbnb6\" (UniqueName: \"kubernetes.io/projected/8312264f-5e72-43fa-babf-d88e31edcccc-kube-api-access-gbnb6\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.342498 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.342267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-webhook-cert\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.342498 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:15.342362 2571 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 09 23:21:15.342498 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:15.342448 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-metrics-certs podName:8312264f-5e72-43fa-babf-d88e31edcccc nodeName:}" failed. No retries permitted until 2026-09-09 23:21:15.84242687 +0000 UTC m=+787.789838590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-metrics-certs") pod "maas-controller-847796847b-rd9zd" (UID: "8312264f-5e72-43fa-babf-d88e31edcccc") : secret "maas-controller-metrics-tls" not found Sep 09 23:21:15.344710 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.344691 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-webhook-cert\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.351358 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.351333 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbnb6\" (UniqueName: \"kubernetes.io/projected/8312264f-5e72-43fa-babf-d88e31edcccc-kube-api-access-gbnb6\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.646037 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.645995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.646240 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.646070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.648571 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.648545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.648687 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.648579 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert\") pod \"maas-controller-c795cf5d5-m9j5f\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:15.847399 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.847363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-metrics-certs\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.849727 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.849700 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8312264f-5e72-43fa-babf-d88e31edcccc-metrics-certs\") pod \"maas-controller-847796847b-rd9zd\" (UID: \"8312264f-5e72-43fa-babf-d88e31edcccc\") " pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:15.864649 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:15.864618 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:16.017746 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:16.017700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:16.132350 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:16.132323 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c795cf5d5-m9j5f"] Sep 09 23:21:16.135146 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:21:16.135104 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595df9aa_c1da_4462_b28e_f8a65a86df44.slice/crio-e8069b0eb31fe26e10d66e35c22b4b2fb169be1afc17cf57b3bd8bab12a92070 WatchSource:0}: Error finding container e8069b0eb31fe26e10d66e35c22b4b2fb169be1afc17cf57b3bd8bab12a92070: Status 404 returned error can't find the container with id e8069b0eb31fe26e10d66e35c22b4b2fb169be1afc17cf57b3bd8bab12a92070 Sep 09 23:21:16.286494 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:16.286468 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-847796847b-rd9zd"] Sep 09 23:21:16.288498 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:21:16.288468 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8312264f_5e72_43fa_babf_d88e31edcccc.slice/crio-af3b735338680e9718f75a3db4f515f120bb2cefefc1653ab9bed97cadc749d1 WatchSource:0}: Error finding container af3b735338680e9718f75a3db4f515f120bb2cefefc1653ab9bed97cadc749d1: Status 404 returned error can't find the container with id af3b735338680e9718f75a3db4f515f120bb2cefefc1653ab9bed97cadc749d1 Sep 09 23:21:16.488643 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:16.488603 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" event={"ID":"595df9aa-c1da-4462-b28e-f8a65a86df44","Type":"ContainerStarted","Data":"e8069b0eb31fe26e10d66e35c22b4b2fb169be1afc17cf57b3bd8bab12a92070"} Sep 09 23:21:16.489700 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:16.489668 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-847796847b-rd9zd" event={"ID":"8312264f-5e72-43fa-babf-d88e31edcccc","Type":"ContainerStarted","Data":"af3b735338680e9718f75a3db4f515f120bb2cefefc1653ab9bed97cadc749d1"} Sep 09 23:21:19.505991 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:19.505953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" event={"ID":"595df9aa-c1da-4462-b28e-f8a65a86df44","Type":"ContainerStarted","Data":"737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d"} Sep 09 23:21:19.506491 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:19.506105 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:19.507553 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:19.507531 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-847796847b-rd9zd" event={"ID":"8312264f-5e72-43fa-babf-d88e31edcccc","Type":"ContainerStarted","Data":"f5c62f2894058dd64ed16c79fef7d6db1b1c8e4d08433fb6b4c7585bb9de4ec8"} Sep 09 23:21:19.507628 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:19.507574 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:19.526124 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:19.526050 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" podStartSLOduration=2.678087728 podStartE2EDuration="5.526032577s" totalImagesPullingTime="2.847944849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:21:19.524442265 +0000 UTC m=+791.471853987" watchObservedRunningTime="2026-09-09 23:21:19.526032577 +0000 UTC m=+791.473444299" Sep 09 23:21:19.543524 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:19.543451 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-847796847b-rd9zd" podStartSLOduration=1.8591801669999999 podStartE2EDuration="4.543430031s" totalImagesPullingTime="2.684249864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:21:19.543198437 +0000 UTC m=+791.490610164" watchObservedRunningTime="2026-09-09 23:21:19.543430031 +0000 UTC m=+791.490841755" Sep 09 23:21:30.515083 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.514978 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:30.515083 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.515049 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-847796847b-rd9zd" Sep 09 23:21:30.583666 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.583631 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c795cf5d5-m9j5f"] Sep 09 23:21:30.583865 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.583814 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" podUID="595df9aa-c1da-4462-b28e-f8a65a86df44" containerName="manager" containerID="cri-o://737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d" gracePeriod=30 Sep 09 23:21:30.825875 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.825840 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:30.884494 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.884456 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert\") pod \"595df9aa-c1da-4462-b28e-f8a65a86df44\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " Sep 09 23:21:30.884494 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.884504 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp68g\" (UniqueName: \"kubernetes.io/projected/595df9aa-c1da-4462-b28e-f8a65a86df44-kube-api-access-tp68g\") pod \"595df9aa-c1da-4462-b28e-f8a65a86df44\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " Sep 09 23:21:30.884766 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.884584 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs\") pod \"595df9aa-c1da-4462-b28e-f8a65a86df44\" (UID: \"595df9aa-c1da-4462-b28e-f8a65a86df44\") " Sep 09 23:21:30.886763 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.886726 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "595df9aa-c1da-4462-b28e-f8a65a86df44" (UID: "595df9aa-c1da-4462-b28e-f8a65a86df44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:21:30.886879 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.886739 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595df9aa-c1da-4462-b28e-f8a65a86df44-kube-api-access-tp68g" (OuterVolumeSpecName: "kube-api-access-tp68g") pod "595df9aa-c1da-4462-b28e-f8a65a86df44" (UID: "595df9aa-c1da-4462-b28e-f8a65a86df44"). InnerVolumeSpecName "kube-api-access-tp68g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 23:21:30.886879 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.886803 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "595df9aa-c1da-4462-b28e-f8a65a86df44" (UID: "595df9aa-c1da-4462-b28e-f8a65a86df44"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 23:21:30.985568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.985530 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-metrics-certs\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:21:30.985568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.985563 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/595df9aa-c1da-4462-b28e-f8a65a86df44-webhook-cert\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:21:30.985568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:30.985572 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp68g\" (UniqueName: \"kubernetes.io/projected/595df9aa-c1da-4462-b28e-f8a65a86df44-kube-api-access-tp68g\") on node \"ip-10-0-140-142.ec2.internal\" DevicePath \"\"" Sep 09 23:21:31.567185 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.567147 2571 generic.go:356] "Generic (PLEG): container finished" podID="595df9aa-c1da-4462-b28e-f8a65a86df44" containerID="737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d" exitCode=0 Sep 09 23:21:31.567185 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.567175 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" Sep 09 23:21:31.567623 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.567213 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" event={"ID":"595df9aa-c1da-4462-b28e-f8a65a86df44","Type":"ContainerDied","Data":"737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d"} Sep 09 23:21:31.567623 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.567264 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c795cf5d5-m9j5f" event={"ID":"595df9aa-c1da-4462-b28e-f8a65a86df44","Type":"ContainerDied","Data":"e8069b0eb31fe26e10d66e35c22b4b2fb169be1afc17cf57b3bd8bab12a92070"} Sep 09 23:21:31.567623 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.567286 2571 scope.go:117] "RemoveContainer" containerID="737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d" Sep 09 23:21:31.576502 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.576485 2571 scope.go:117] "RemoveContainer" containerID="737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d" Sep 09 23:21:31.576781 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:21:31.576762 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d\": container with ID starting with 737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d not found: ID does not exist" containerID="737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d" Sep 09 23:21:31.576826 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.576791 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d"} err="failed to get container status \"737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d\": rpc error: code = NotFound desc = could not find container \"737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d\": container with ID starting with 737df4e048800223592778cd1bc3b4e21fca716b962cc69f2111044f9d0a867d not found: ID does not exist" Sep 09 23:21:31.591887 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.591853 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c795cf5d5-m9j5f"] Sep 09 23:21:31.595883 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:31.595851 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c795cf5d5-m9j5f"] Sep 09 23:21:32.549822 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:32.549789 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="595df9aa-c1da-4462-b28e-f8a65a86df44" path="/var/lib/kubelet/pods/595df9aa-c1da-4462-b28e-f8a65a86df44/volumes" Sep 09 23:21:52.533957 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.533914 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-665f4d5f8-79lls"] Sep 09 23:21:52.534563 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.534386 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="595df9aa-c1da-4462-b28e-f8a65a86df44" containerName="manager" Sep 09 23:21:52.534563 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.534404 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="595df9aa-c1da-4462-b28e-f8a65a86df44" containerName="manager" Sep 09 23:21:52.534563 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.534486 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="595df9aa-c1da-4462-b28e-f8a65a86df44" containerName="manager" Sep 09 23:21:52.540706 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.540682 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.543785 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.543749 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 23:21:52.545178 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.545156 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 23:21:52.545312 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.545168 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-m7vfb\"" Sep 09 23:21:52.551099 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.551068 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-665f4d5f8-79lls"] Sep 09 23:21:52.656875 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.656834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aedcced4-5847-423f-9e86-faaa237458b4-metrics-certs\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.657042 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.656890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aedcced4-5847-423f-9e86-faaa237458b4-maas-api-tls\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.657042 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.656943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcced4-5847-423f-9e86-faaa237458b4-openshift-service-ca\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.657042 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.656975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7kc\" (UniqueName: \"kubernetes.io/projected/aedcced4-5847-423f-9e86-faaa237458b4-kube-api-access-6m7kc\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.757839 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.757795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aedcced4-5847-423f-9e86-faaa237458b4-metrics-certs\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.758016 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.757854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aedcced4-5847-423f-9e86-faaa237458b4-maas-api-tls\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.758016 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.757877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcced4-5847-423f-9e86-faaa237458b4-openshift-service-ca\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.758016 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.757908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7kc\" (UniqueName: \"kubernetes.io/projected/aedcced4-5847-423f-9e86-faaa237458b4-kube-api-access-6m7kc\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.758676 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.758645 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcced4-5847-423f-9e86-faaa237458b4-openshift-service-ca\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.760289 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.760259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aedcced4-5847-423f-9e86-faaa237458b4-metrics-certs\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.760466 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.760445 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aedcced4-5847-423f-9e86-faaa237458b4-maas-api-tls\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.766328 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.766297 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7kc\" (UniqueName: \"kubernetes.io/projected/aedcced4-5847-423f-9e86-faaa237458b4-kube-api-access-6m7kc\") pod \"maas-api-665f4d5f8-79lls\" (UID: \"aedcced4-5847-423f-9e86-faaa237458b4\") " pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:52.853346 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:52.853310 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:53.127456 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:53.127429 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-665f4d5f8-79lls"] Sep 09 23:21:53.129238 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:21:53.129212 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedcced4_5847_423f_9e86_faaa237458b4.slice/crio-d0623838711629d64acf0130c4d0b4cf90d15ce6a218c1248d796d40de406f24 WatchSource:0}: Error finding container d0623838711629d64acf0130c4d0b4cf90d15ce6a218c1248d796d40de406f24: Status 404 returned error can't find the container with id d0623838711629d64acf0130c4d0b4cf90d15ce6a218c1248d796d40de406f24 Sep 09 23:21:53.671550 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:53.671504 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" event={"ID":"aedcced4-5847-423f-9e86-faaa237458b4","Type":"ContainerStarted","Data":"d0623838711629d64acf0130c4d0b4cf90d15ce6a218c1248d796d40de406f24"} Sep 09 23:21:55.681242 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:55.681203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" event={"ID":"aedcced4-5847-423f-9e86-faaa237458b4","Type":"ContainerStarted","Data":"619b6fa5392e99bdeec46eb84bc15b199a1b8b2cfecb623cb387531fd4e6b0cb"} Sep 09 23:21:55.681714 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:55.681349 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:21:55.699992 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:21:55.699938 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" podStartSLOduration=1.6184375009999998 podStartE2EDuration="3.699923132s" totalImagesPullingTime="2.081485631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:21:55.698680373 +0000 UTC m=+827.646092096" watchObservedRunningTime="2026-09-09 23:21:55.699923132 +0000 UTC m=+827.647334856" Sep 09 23:22:01.691677 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:01.691645 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-665f4d5f8-79lls" Sep 09 23:22:02.717245 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:02.717205 2571 generic.go:356] "Generic (PLEG): container finished" podID="751437d2-bef6-4c18-b471-6514a7dcb80b" containerID="0c28ced0aff11746940f3f1f520fb36f4186e7211575fdb1c97f01ae8f503234" exitCode=0 Sep 09 23:22:02.717764 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:02.717278 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" event={"ID":"751437d2-bef6-4c18-b471-6514a7dcb80b","Type":"ContainerDied","Data":"0c28ced0aff11746940f3f1f520fb36f4186e7211575fdb1c97f01ae8f503234"} Sep 09 23:22:02.717764 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:02.717671 2571 scope.go:117] "RemoveContainer" containerID="0c28ced0aff11746940f3f1f520fb36f4186e7211575fdb1c97f01ae8f503234" Sep 09 23:22:02.718740 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:02.718708 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:22:02.718816 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:02.718791 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:22:02.718875 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:02.718838 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 09 23:22:03.728046 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:03.728001 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" event={"ID":"751437d2-bef6-4c18-b471-6514a7dcb80b","Type":"ContainerStarted","Data":"6bd10244c1ac4c07a31b442e3f511ed585e3a76ec8545b8f8ccddb183ef9022b"} Sep 09 23:22:03.792837 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:03.792792 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:22:03.793093 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:03.793073 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nhpdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 09 23:22:03.793190 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:03.793148 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" podUID="751437d2-bef6-4c18-b471-6514a7dcb80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 09 23:22:04.792465 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:04.792429 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nhpdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 09 23:22:04.792851 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:04.792507 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" podUID="751437d2-bef6-4c18-b471-6514a7dcb80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 09 23:22:05.792601 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:05.792559 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nhpdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 09 23:22:05.792978 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:05.792626 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" podUID="751437d2-bef6-4c18-b471-6514a7dcb80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 09 23:22:06.792747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:06.792704 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nhpdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 09 23:22:06.793162 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:06.792763 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" podUID="751437d2-bef6-4c18-b471-6514a7dcb80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 09 23:22:07.792911 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:07.792878 2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nhpdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Sep 09 23:22:07.793279 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:07.792930 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" podUID="751437d2-bef6-4c18-b471-6514a7dcb80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Sep 09 23:22:08.796460 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:08.796428 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:22:08.796950 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:08.796707 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:22:08.797321 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:08.797305 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nhpdc" Sep 09 23:22:27.394167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.394133 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5"] Sep 09 23:22:27.397221 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.397195 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.400259 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.400227 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 23:22:27.400401 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.400265 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 09 23:22:27.400401 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.400387 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n6sp2\"" Sep 09 23:22:27.401792 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.401773 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 23:22:27.406365 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.406331 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5"] Sep 09 23:22:27.564284 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.564251 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.564463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.564296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.564463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.564329 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30e8893-7115-40df-9db0-d2722d5f81ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.564463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.564375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.564463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.564421 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.564463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.564451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbsx\" (UniqueName: \"kubernetes.io/projected/d30e8893-7115-40df-9db0-d2722d5f81ec-kube-api-access-kmbsx\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.665914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.665811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbsx\" (UniqueName: \"kubernetes.io/projected/d30e8893-7115-40df-9db0-d2722d5f81ec-kube-api-access-kmbsx\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.665914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.665868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.665914 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.665913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.666227 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.665948 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30e8893-7115-40df-9db0-d2722d5f81ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.666227 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.665973 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.666227 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.666168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.666403 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.666386 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-home\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.666495 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.666467 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-tmp-dir\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.666602 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.666523 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-model-cache\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.668339 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.668316 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d30e8893-7115-40df-9db0-d2722d5f81ec-dshm\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.668718 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.668696 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d30e8893-7115-40df-9db0-d2722d5f81ec-tls-certs\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.675942 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.675881 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbsx\" (UniqueName: \"kubernetes.io/projected/d30e8893-7115-40df-9db0-d2722d5f81ec-kube-api-access-kmbsx\") pod \"e2e-distinct-simulated-kserve-6685d86975-8lqv5\" (UID: \"d30e8893-7115-40df-9db0-d2722d5f81ec\") " pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.708337 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.708289 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:27.976855 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:27.976827 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5"] Sep 09 23:22:27.978698 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:22:27.978664 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30e8893_7115_40df_9db0_d2722d5f81ec.slice/crio-5ccb2f2c9d259914c6d8801d3e5ba746b8564a4ac3f72788d9c93893f39ef4d9 WatchSource:0}: Error finding container 5ccb2f2c9d259914c6d8801d3e5ba746b8564a4ac3f72788d9c93893f39ef4d9: Status 404 returned error can't find the container with id 5ccb2f2c9d259914c6d8801d3e5ba746b8564a4ac3f72788d9c93893f39ef4d9 Sep 09 23:22:28.845833 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:28.845795 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerStarted","Data":"5ccb2f2c9d259914c6d8801d3e5ba746b8564a4ac3f72788d9c93893f39ef4d9"} Sep 09 23:22:29.851856 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:29.851817 2571 generic.go:356] "Generic (PLEG): container finished" podID="d30e8893-7115-40df-9db0-d2722d5f81ec" containerID="89850601449f4fc52cd56ca12753870fea4138de01d6b70cf4993d02bb4acc0f" exitCode=0 Sep 09 23:22:29.852333 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:29.851902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerDied","Data":"89850601449f4fc52cd56ca12753870fea4138de01d6b70cf4993d02bb4acc0f"} Sep 09 23:22:29.852333 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:29.852146 2571 scope.go:117] "RemoveContainer" containerID="89850601449f4fc52cd56ca12753870fea4138de01d6b70cf4993d02bb4acc0f" Sep 09 23:22:30.858859 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:30.858826 2571 generic.go:356] "Generic (PLEG): container finished" podID="d30e8893-7115-40df-9db0-d2722d5f81ec" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" exitCode=0 Sep 09 23:22:30.859296 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:30.858915 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerDied","Data":"82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973"} Sep 09 23:22:30.859296 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:30.858969 2571 scope.go:117] "RemoveContainer" containerID="89850601449f4fc52cd56ca12753870fea4138de01d6b70cf4993d02bb4acc0f" Sep 09 23:22:30.859296 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:30.859255 2571 scope.go:117] "RemoveContainer" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" Sep 09 23:22:30.859468 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:22:30.859450 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:22:31.865569 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:31.865542 2571 scope.go:117] "RemoveContainer" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" Sep 09 23:22:31.865995 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:22:31.865714 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:22:37.708813 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:37.708775 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:37.709249 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:37.708858 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:37.709249 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:37.708906 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:37.709249 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:37.708923 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:37.709352 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:37.709266 2571 scope.go:117] "RemoveContainer" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" Sep 09 23:22:37.709456 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:22:37.709435 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:22:37.891238 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:37.891209 2571 scope.go:117] "RemoveContainer" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" Sep 09 23:22:37.891397 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:22:37.891376 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:22:51.544608 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:51.544575 2571 scope.go:117] "RemoveContainer" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" Sep 09 23:22:51.952479 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:51.952437 2571 generic.go:356] "Generic (PLEG): container finished" podID="d30e8893-7115-40df-9db0-d2722d5f81ec" containerID="ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888" exitCode=0 Sep 09 23:22:51.952641 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:51.952512 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerDied","Data":"ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888"} Sep 09 23:22:51.952641 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:51.952571 2571 scope.go:117] "RemoveContainer" containerID="82c507681e28431533a75e624ccece413990cb5690dfeacf639e7735f649d973" Sep 09 23:22:51.952937 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:51.952907 2571 scope.go:117] "RemoveContainer" containerID="ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888" Sep 09 23:22:51.953152 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:22:51.953102 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:22:54.722603 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.722571 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn"] Sep 09 23:22:54.725559 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.725539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.728758 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.728734 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 09 23:22:54.734086 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.734058 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn"] Sep 09 23:22:54.793564 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.793522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.793739 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.793578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.793739 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.793693 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.793872 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.793764 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59b83c4-b31d-4428-9c92-2855d8d57327-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.793872 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.793804 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mdz\" (UniqueName: \"kubernetes.io/projected/f59b83c4-b31d-4428-9c92-2855d8d57327-kube-api-access-l5mdz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.793872 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.793833 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.894532 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.894494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.894750 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.894542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.894750 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.894582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.894750 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.894606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59b83c4-b31d-4428-9c92-2855d8d57327-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.894750 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.894649 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mdz\" (UniqueName: \"kubernetes.io/projected/f59b83c4-b31d-4428-9c92-2855d8d57327-kube-api-access-l5mdz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.895205 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.895051 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.895205 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.895053 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.895205 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.895166 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.895450 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.895288 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.897285 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.897256 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f59b83c4-b31d-4428-9c92-2855d8d57327-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.897548 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.897519 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f59b83c4-b31d-4428-9c92-2855d8d57327-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:54.903077 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:54.903056 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mdz\" (UniqueName: \"kubernetes.io/projected/f59b83c4-b31d-4428-9c92-2855d8d57327-kube-api-access-l5mdz\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn\" (UID: \"f59b83c4-b31d-4428-9c92-2855d8d57327\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:55.055440 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:55.055406 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:22:55.324167 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:55.324134 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn"] Sep 09 23:22:55.326472 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:22:55.326442 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59b83c4_b31d_4428_9c92_2855d8d57327.slice/crio-90d4212ed3bdf1da5a6f998c5aaa508f234192cc5b13456c9ea037dedcdf9486 WatchSource:0}: Error finding container 90d4212ed3bdf1da5a6f998c5aaa508f234192cc5b13456c9ea037dedcdf9486: Status 404 returned error can't find the container with id 90d4212ed3bdf1da5a6f998c5aaa508f234192cc5b13456c9ea037dedcdf9486 Sep 09 23:22:55.974664 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:55.974266 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" event={"ID":"f59b83c4-b31d-4428-9c92-2855d8d57327","Type":"ContainerStarted","Data":"519af5bb3d8bbbebd084cf856e894d542055a446df2d31eead62a9a41109e8bb"} Sep 09 23:22:55.974664 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:55.974335 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" event={"ID":"f59b83c4-b31d-4428-9c92-2855d8d57327","Type":"ContainerStarted","Data":"90d4212ed3bdf1da5a6f998c5aaa508f234192cc5b13456c9ea037dedcdf9486"} Sep 09 23:22:55.992478 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:55.992428 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" podStartSLOduration=1.827737601 podStartE2EDuration="1.992410559s" totalImagesPullingTime="164.672958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 23:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:22:55.99230009 +0000 UTC m=+887.939711814" watchObservedRunningTime="2026-09-09 23:22:55.992410559 +0000 UTC m=+887.939822281" Sep 09 23:22:57.709324 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:57.709284 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:57.709768 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:57.709346 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:57.709768 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:57.709363 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:57.709768 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:57.709377 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:22:57.709880 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:22:57.709814 2571 scope.go:117] "RemoveContainer" containerID="ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888" Sep 09 23:22:57.710009 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:22:57.709988 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:23:05.056394 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:05.056305 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:23:05.056394 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:05.056387 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:23:05.068872 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:05.068843 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:23:06.028768 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:06.028733 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn" Sep 09 23:23:09.544910 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:09.544874 2571 scope.go:117] "RemoveContainer" containerID="ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888" Sep 09 23:23:09.545354 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:09.545049 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:23:21.544310 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:21.544277 2571 scope.go:117] "RemoveContainer" containerID="ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888" Sep 09 23:23:22.088453 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:22.088363 2571 generic.go:356] "Generic (PLEG): container finished" podID="d30e8893-7115-40df-9db0-d2722d5f81ec" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" exitCode=0 Sep 09 23:23:22.088453 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:22.088435 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerDied","Data":"f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220"} Sep 09 23:23:22.088628 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:22.088494 2571 scope.go:117] "RemoveContainer" containerID="ff05a51c93c697d58416dc3bfe47780e9c3b36c38966ca3e99589aa2570d6888" Sep 09 23:23:22.088855 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:22.088832 2571 scope.go:117] "RemoveContainer" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" Sep 09 23:23:22.089022 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:22.089004 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:23:25.112759 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.112720 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd"] Sep 09 23:23:25.117816 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.117792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.120699 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.120675 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 09 23:23:25.127058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.127033 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd"] Sep 09 23:23:25.175648 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.175606 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.175835 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.175659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.175835 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.175729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.175835 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.175802 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.175835 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.175830 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4t92\" (UniqueName: \"kubernetes.io/projected/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-kube-api-access-d4t92\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.176050 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.175887 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.276713 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.276670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.276909 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.276725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.276909 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.276752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.276909 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.276777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.276909 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.276806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.277154 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.276936 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4t92\" (UniqueName: \"kubernetes.io/projected/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-kube-api-access-d4t92\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.277305 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.277276 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-home\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.277372 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.277325 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.277372 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.277358 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.279029 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.278999 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-dshm\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.279412 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.279388 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.286550 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.286521 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4t92\" (UniqueName: \"kubernetes.io/projected/755c8f99-03be-4c2f-a08f-c6b2a5d863e2-kube-api-access-d4t92\") pod \"facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd\" (UID: \"755c8f99-03be-4c2f-a08f-c6b2a5d863e2\") " pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.428830 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.428734 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:25.698223 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:25.698196 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd"] Sep 09 23:23:25.700210 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:23:25.700177 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755c8f99_03be_4c2f_a08f_c6b2a5d863e2.slice/crio-d96728c228964b343dd231e1662085287ebee4bbd13d28d314b9500e99afb47b WatchSource:0}: Error finding container d96728c228964b343dd231e1662085287ebee4bbd13d28d314b9500e99afb47b: Status 404 returned error can't find the container with id d96728c228964b343dd231e1662085287ebee4bbd13d28d314b9500e99afb47b Sep 09 23:23:26.115341 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:26.115302 2571 generic.go:356] "Generic (PLEG): container finished" podID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" containerID="813071024b13253a6ba98cea22a256b0dd46ac2c73bfc889c19a41099f2227f1" exitCode=0 Sep 09 23:23:26.115341 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:26.115342 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerDied","Data":"813071024b13253a6ba98cea22a256b0dd46ac2c73bfc889c19a41099f2227f1"} Sep 09 23:23:26.115809 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:26.115385 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerStarted","Data":"d96728c228964b343dd231e1662085287ebee4bbd13d28d314b9500e99afb47b"} Sep 09 23:23:26.115809 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:26.115625 2571 scope.go:117] "RemoveContainer" containerID="813071024b13253a6ba98cea22a256b0dd46ac2c73bfc889c19a41099f2227f1" Sep 09 23:23:27.122558 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.122523 2571 generic.go:356] "Generic (PLEG): container finished" podID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" containerID="bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b" exitCode=0 Sep 09 23:23:27.123003 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.122563 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerDied","Data":"bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b"} Sep 09 23:23:27.123003 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.122638 2571 scope.go:117] "RemoveContainer" containerID="813071024b13253a6ba98cea22a256b0dd46ac2c73bfc889c19a41099f2227f1" Sep 09 23:23:27.123003 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.122891 2571 scope.go:117] "RemoveContainer" containerID="bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b" Sep 09 23:23:27.123205 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:27.123090 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:23:27.708570 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.708521 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:23:27.708746 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.708603 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:23:27.708746 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.708623 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:23:27.708746 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.708638 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:23:27.708989 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:27.708976 2571 scope.go:117] "RemoveContainer" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" Sep 09 23:23:27.709202 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:27.709178 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:23:28.129241 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:28.129217 2571 scope.go:117] "RemoveContainer" containerID="bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b" Sep 09 23:23:28.129646 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:28.129381 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:23:35.429575 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:35.429523 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:35.429949 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:35.429602 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:35.429949 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:35.429621 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:35.429949 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:35.429636 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:35.430064 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:35.430034 2571 scope.go:117] "RemoveContainer" containerID="bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b" Sep 09 23:23:35.430279 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:35.430259 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 10s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:23:40.544616 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:40.544578 2571 scope.go:117] "RemoveContainer" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" Sep 09 23:23:40.545065 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:40.544776 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:23:46.544909 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:46.544873 2571 scope.go:117] "RemoveContainer" containerID="bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b" Sep 09 23:23:47.216902 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:47.216819 2571 generic.go:356] "Generic (PLEG): container finished" podID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" containerID="165d84d26a8a4808b815deb76344ed3b89ae87db477b7dd7308d00aa54dd63c7" exitCode=0 Sep 09 23:23:47.216902 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:47.216886 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerDied","Data":"165d84d26a8a4808b815deb76344ed3b89ae87db477b7dd7308d00aa54dd63c7"} Sep 09 23:23:47.217073 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:47.216939 2571 scope.go:117] "RemoveContainer" containerID="bdb49f953abfef384b8697cb81e008154af173d5ce915b65fa676fe1ca8c565b" Sep 09 23:23:47.217370 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:47.217334 2571 scope.go:117] "RemoveContainer" containerID="165d84d26a8a4808b815deb76344ed3b89ae87db477b7dd7308d00aa54dd63c7" Sep 09 23:23:47.217594 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:47.217575 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:23:52.545027 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:52.544991 2571 scope.go:117] "RemoveContainer" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" Sep 09 23:23:52.545530 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:52.545194 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:23:55.429854 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:55.429809 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:55.430354 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:55.429888 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:55.430354 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:55.429911 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:55.430354 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:55.429932 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:23:55.430511 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:23:55.430472 2571 scope.go:117] "RemoveContainer" containerID="165d84d26a8a4808b815deb76344ed3b89ae87db477b7dd7308d00aa54dd63c7" Sep 09 23:23:55.430692 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:23:55.430667 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:24:03.544649 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:03.544615 2571 scope.go:117] "RemoveContainer" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" Sep 09 23:24:04.295409 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:04.295374 2571 generic.go:356] "Generic (PLEG): container finished" podID="d30e8893-7115-40df-9db0-d2722d5f81ec" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" exitCode=0 Sep 09 23:24:04.295409 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:04.295414 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerDied","Data":"20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8"} Sep 09 23:24:04.295670 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:04.295466 2571 scope.go:117] "RemoveContainer" containerID="f331e56096a6da75c3b81168ad9e289c185e9c143e7d226440e30ba0accf4220" Sep 09 23:24:04.295864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:04.295845 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:24:04.296076 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:04.296056 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:24:07.709109 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:07.709056 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:24:07.709568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:07.709161 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:24:07.709568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:07.709177 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:24:07.709568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:07.709191 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:24:07.709568 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:07.709529 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:24:07.709708 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:07.709691 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:24:08.546718 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:08.546684 2571 scope.go:117] "RemoveContainer" containerID="165d84d26a8a4808b815deb76344ed3b89ae87db477b7dd7308d00aa54dd63c7" Sep 09 23:24:09.320722 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:09.320689 2571 generic.go:356] "Generic (PLEG): container finished" podID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" exitCode=0 Sep 09 23:24:09.321184 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:09.320741 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerDied","Data":"5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8"} Sep 09 23:24:09.321184 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:09.320788 2571 scope.go:117] "RemoveContainer" containerID="165d84d26a8a4808b815deb76344ed3b89ae87db477b7dd7308d00aa54dd63c7" Sep 09 23:24:09.321184 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:09.321147 2571 scope.go:117] "RemoveContainer" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" Sep 09 23:24:09.321358 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:09.321337 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:24:15.429722 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:15.429677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:24:15.430176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:15.429745 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:24:15.430176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:15.429761 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:24:15.430176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:15.429775 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:24:15.430176 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:15.430095 2571 scope.go:117] "RemoveContainer" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" Sep 09 23:24:15.430317 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:15.430279 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:24:22.544584 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:22.544548 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:24:22.544972 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:22.544730 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:24:30.544415 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:30.544389 2571 scope.go:117] "RemoveContainer" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" Sep 09 23:24:30.544794 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:30.544589 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:24:33.544828 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:33.544789 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:24:33.545269 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:33.544976 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:24:42.549849 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:42.549818 2571 scope.go:117] "RemoveContainer" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" Sep 09 23:24:42.550350 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:42.549990 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:24:45.544619 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:45.544578 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:24:45.545082 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:45.544809 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:24:55.544247 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:55.544209 2571 scope.go:117] "RemoveContainer" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" Sep 09 23:24:55.544828 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:55.544594 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 23:24:56.534219 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:56.534181 2571 generic.go:356] "Generic (PLEG): container finished" podID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" exitCode=0 Sep 09 23:24:56.534414 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:56.534250 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerDied","Data":"deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c"} Sep 09 23:24:56.534414 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:56.534314 2571 scope.go:117] "RemoveContainer" containerID="5927e482e6ae14d3c04def8af5f88497439b11821f9cf72b41afc3c4ef8aabf8" Sep 09 23:24:56.534686 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:56.534671 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:24:56.534910 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:56.534891 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:24:59.544218 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:24:59.544185 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:24:59.544598 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:24:59.544376 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:25:05.429256 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:05.429211 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:25:05.429824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:05.429281 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:25:05.429824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:05.429296 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:25:05.429824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:05.429311 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:25:05.429824 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:05.429664 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:25:05.429985 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:05.429847 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:25:13.544327 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:13.544292 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:25:13.544783 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:13.544465 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:25:17.544381 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:17.544346 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:25:17.544766 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:17.544527 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:25:25.544229 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:25.544194 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:25:26.668343 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:26.668308 2571 generic.go:356] "Generic (PLEG): container finished" podID="d30e8893-7115-40df-9db0-d2722d5f81ec" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" exitCode=0 Sep 09 23:25:26.668808 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:26.668381 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" event={"ID":"d30e8893-7115-40df-9db0-d2722d5f81ec","Type":"ContainerDied","Data":"3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609"} Sep 09 23:25:26.668808 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:26.668439 2571 scope.go:117] "RemoveContainer" containerID="20188a6477930cd74bf658b252a72579677f70d9bdd2132ca9ce2af3664802b8" Sep 09 23:25:26.668917 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:26.668847 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:25:26.669068 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:26.669047 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:25:27.708432 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:27.708384 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:25:27.708858 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:27.708468 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:25:27.708858 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:27.708492 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:25:27.708858 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:27.708509 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" Sep 09 23:25:27.708964 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:27.708881 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:25:27.709072 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:27.709052 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:25:31.544677 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:31.544641 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:25:31.545177 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:31.544816 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:25:41.544528 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:41.544489 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:25:41.544945 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:41.544684 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:25:43.544861 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:43.544824 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:25:43.545337 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:43.545013 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:25:54.544952 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:54.544916 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:25:54.545447 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:54.545097 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:25:58.547864 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:25:58.547833 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:25:58.548247 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:25:58.548011 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:26:08.547424 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:08.547324 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:26:08.547996 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:08.547561 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:26:13.544652 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:13.544616 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:26:13.545058 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:13.544783 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:26:23.544585 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:23.544549 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:26:23.545063 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:23.544714 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:26:28.545979 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:28.545948 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:26:28.949537 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:28.949497 2571 generic.go:356] "Generic (PLEG): container finished" podID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" exitCode=0 Sep 09 23:26:28.949722 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:28.949558 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" event={"ID":"755c8f99-03be-4c2f-a08f-c6b2a5d863e2","Type":"ContainerDied","Data":"72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483"} Sep 09 23:26:28.949722 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:28.949609 2571 scope.go:117] "RemoveContainer" containerID="deb2911b52dd19d567d279f6344901f2d517750720411dbb63177679c8b2a17c" Sep 09 23:26:28.949979 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:28.949958 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:26:28.950247 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:28.950222 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:26:35.429858 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:35.429808 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:26:35.430284 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:35.429885 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:26:35.430284 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:35.429901 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:26:35.430284 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:35.429917 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" Sep 09 23:26:35.430393 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:35.430295 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:26:35.430480 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:35.430461 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:26:35.544359 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:35.544323 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:26:35.544521 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:35.544507 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:26:48.546901 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:48.546870 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:26:48.547309 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:48.547056 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:26:50.544407 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:26:50.544371 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:26:50.544787 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:26:50.544560 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:27:01.544756 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:01.544720 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:27:01.545278 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:01.544913 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:27:02.544747 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:02.544695 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:27:02.546816 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:02.544902 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:27:05.904635 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:05.904605 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_755c8f99-03be-4c2f-a08f-c6b2a5d863e2/main/5.log" Sep 09 23:27:06.020814 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:06.020784 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_755c8f99-03be-4c2f-a08f-c6b2a5d863e2/main/5.log" Sep 09 23:27:12.604713 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:12.604667 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-847796847b-rd9zd_8312264f-5e72-43fa-babf-d88e31edcccc/manager/0.log" Sep 09 23:27:12.714967 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:12.714934 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n2htd_e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d/server/0.log" Sep 09 23:27:12.833133 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:12.833090 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kcs88_2b53de38-ad45-40a1-956d-4bfedee7339c/manager/0.log" Sep 09 23:27:13.207728 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:13.207696 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68fd95c47c-zss8z_c4365b36-0d59-4e67-9c77-311cec6242fd/copy-manifests/0.log" Sep 09 23:27:13.214626 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:13.214599 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68fd95c47c-zss8z_c4365b36-0d59-4e67-9c77-311cec6242fd/manager/0.log" Sep 09 23:27:13.561708 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:13.561665 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-665f4d5f8-79lls_aedcced4-5847-423f-9e86-faaa237458b4/maas-api/0.log" Sep 09 23:27:13.676878 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:13.676843 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gncql_6661d673-1114-4a2d-89d3-dcbc79d90b56/postgres/0.log" Sep 09 23:27:14.544242 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:14.544206 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:27:14.544492 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:14.544469 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:27:14.564976 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:14.564941 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-hp6cv_d39c290f-5941-4870-abbc-e4ea35964671/manager/0.log" Sep 09 23:27:15.491921 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:15.491889 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6_39309e3d-3900-4ecd-9bd9-dfab92e05360/istio-proxy/0.log" Sep 09 23:27:15.544899 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:15.544863 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:27:15.545084 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:15.545068 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:27:15.615763 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:15.615726 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fj64c_8d4da4b1-7550-42cd-a17e-e1390c57beba/discovery/0.log" Sep 09 23:27:15.838566 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:15.838536 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576f68f9bc-wsjmn_07f35310-390a-48fd-b29f-9c1d08b3dd81/kube-auth-proxy/0.log" Sep 09 23:27:15.953252 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:15.953218 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/1.log" Sep 09 23:27:16.774186 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:16.774156 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-6685d86975-8lqv5_d30e8893-7115-40df-9db0-d2722d5f81ec/main/5.log" Sep 09 23:27:16.888655 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:16.888623 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-lzvxn_f59b83c4-b31d-4428-9c92-2855d8d57327/main/0.log" Sep 09 23:27:17.261670 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:17.261636 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_755c8f99-03be-4c2f-a08f-c6b2a5d863e2/main/5.log" Sep 09 23:27:26.424340 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:26.424304 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qp8cl_b5bffd7d-c4a7-4d1e-a5f4-a808ab84e0ea/global-pull-secret-syncer/0.log" Sep 09 23:27:26.528363 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:26.528323 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sv6q5_381e539a-f46d-4ce6-b093-8192a74feb89/konnectivity-agent/0.log" Sep 09 23:27:26.594467 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:26.594437 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-142.ec2.internal_18d764b0ed3aa25d3bd2d54df9d49711/haproxy/0.log" Sep 09 23:27:28.546605 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:28.546564 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:27:28.547081 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:28.546796 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:27:29.543970 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:29.543936 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:27:29.544179 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:29.544107 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:27:30.790017 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:30.789939 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-hp6cv_d39c290f-5941-4870-abbc-e4ea35964671/manager/0.log" Sep 09 23:27:33.043557 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:33.043528 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7tn22_d5e895e5-ef52-4938-8fcf-e46232bc95a7/node-exporter/0.log" Sep 09 23:27:33.064068 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:33.064040 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7tn22_d5e895e5-ef52-4938-8fcf-e46232bc95a7/kube-rbac-proxy/0.log" Sep 09 23:27:33.084058 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:33.084018 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7tn22_d5e895e5-ef52-4938-8fcf-e46232bc95a7/init-textfile/0.log" Sep 09 23:27:35.235903 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:35.235876 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-cjdml_9f2a5c74-93c8-486e-a372-c4223fe2ac2f/networking-console-plugin/0.log" Sep 09 23:27:36.319369 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.319334 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2"] Sep 09 23:27:36.322792 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.322770 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.325782 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.325754 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h2lnk\"/\"kube-root-ca.crt\"" Sep 09 23:27:36.327086 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.327068 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h2lnk\"/\"default-dockercfg-2mqrr\"" Sep 09 23:27:36.327214 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.327090 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h2lnk\"/\"openshift-service-ca.crt\"" Sep 09 23:27:36.331846 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.331821 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2"] Sep 09 23:27:36.484264 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.484217 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-lib-modules\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.484455 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.484308 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpfn\" (UniqueName: \"kubernetes.io/projected/b1172e3e-5180-4249-8e21-5f6ef766aa8d-kube-api-access-qgpfn\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.484455 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.484349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-podres\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.484455 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.484417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-sys\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.484559 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.484499 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-proc\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585495 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-proc\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585642 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585504 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-lib-modules\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585642 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585535 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-proc\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585642 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpfn\" (UniqueName: \"kubernetes.io/projected/b1172e3e-5180-4249-8e21-5f6ef766aa8d-kube-api-access-qgpfn\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585742 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-podres\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585742 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-sys\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585742 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585708 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-lib-modules\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585837 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585768 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-sys\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.585837 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.585799 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1172e3e-5180-4249-8e21-5f6ef766aa8d-podres\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.594045 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.594003 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpfn\" (UniqueName: \"kubernetes.io/projected/b1172e3e-5180-4249-8e21-5f6ef766aa8d-kube-api-access-qgpfn\") pod \"perf-node-gather-daemonset-jhxr2\" (UID: \"b1172e3e-5180-4249-8e21-5f6ef766aa8d\") " pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.633003 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.632943 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:36.903493 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:36.903460 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2"] Sep 09 23:27:36.905076 ip-10-0-140-142 kubenswrapper[2571]: W0909 23:27:36.905047 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1172e3e_5180_4249_8e21_5f6ef766aa8d.slice/crio-3753e16b32c4c826a5d1824f974bbc719b7261167d47d0129a1b941d3bc126db WatchSource:0}: Error finding container 3753e16b32c4c826a5d1824f974bbc719b7261167d47d0129a1b941d3bc126db: Status 404 returned error can't find the container with id 3753e16b32c4c826a5d1824f974bbc719b7261167d47d0129a1b941d3bc126db Sep 09 23:27:37.252594 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:37.252496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" event={"ID":"b1172e3e-5180-4249-8e21-5f6ef766aa8d","Type":"ContainerStarted","Data":"57fb55226ef88901f44163404d5a93cdd4e56a15e32e29c8d0e0d0075b7f6059"} Sep 09 23:27:37.252594 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:37.252555 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" event={"ID":"b1172e3e-5180-4249-8e21-5f6ef766aa8d","Type":"ContainerStarted","Data":"3753e16b32c4c826a5d1824f974bbc719b7261167d47d0129a1b941d3bc126db"} Sep 09 23:27:37.252777 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:37.252596 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:37.272312 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:37.272255 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" podStartSLOduration=1.272237337 podStartE2EDuration="1.272237337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 23:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 23:27:37.271163737 +0000 UTC m=+1169.218575463" watchObservedRunningTime="2026-09-09 23:27:37.272237337 +0000 UTC m=+1169.219649061" Sep 09 23:27:38.153447 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:38.153406 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hgdps_99fad466-3949-46f2-b9cf-384cc72fc275/dns/0.log" Sep 09 23:27:38.172106 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:38.172079 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hgdps_99fad466-3949-46f2-b9cf-384cc72fc275/kube-rbac-proxy/0.log" Sep 09 23:27:38.279555 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:38.279523 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tg9zt_2ae6772e-e29f-49ed-b872-0c2f6b30ac90/dns-node-resolver/0.log" Sep 09 23:27:38.852953 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:38.852921 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v7v25_86cd631b-4036-4398-b0a6-63d6c08bb6b8/node-ca/0.log" Sep 09 23:27:39.731685 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:39.731651 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6_39309e3d-3900-4ecd-9bd9-dfab92e05360/istio-proxy/0.log" Sep 09 23:27:39.792887 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:39.792851 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fj64c_8d4da4b1-7550-42cd-a17e-e1390c57beba/discovery/0.log" Sep 09 23:27:39.837638 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:39.837608 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576f68f9bc-wsjmn_07f35310-390a-48fd-b29f-9c1d08b3dd81/kube-auth-proxy/0.log" Sep 09 23:27:39.864463 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:39.864424 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/1.log" Sep 09 23:27:39.865000 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:39.864983 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/0.log" Sep 09 23:27:40.506566 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:40.506534 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7vdvv_08b1f98e-1a5b-4d04-ab10-570c9584e1bd/serve-healthcheck-canary/0.log" Sep 09 23:27:40.992029 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:40.991991 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lrfw_ce94d45c-9b72-48ec-8019-2f744435478e/kube-rbac-proxy/0.log" Sep 09 23:27:41.011451 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:41.011421 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lrfw_ce94d45c-9b72-48ec-8019-2f744435478e/exporter/0.log" Sep 09 23:27:41.033177 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:41.033146 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lrfw_ce94d45c-9b72-48ec-8019-2f744435478e/extractor/0.log" Sep 09 23:27:41.544784 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:41.544750 2571 scope.go:117] "RemoveContainer" containerID="72b7216e707b7dd51f718097064596f81183be8231732e1f678e86f233114483" Sep 09 23:27:41.545002 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:41.544927 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd_llm(755c8f99-03be-4c2f-a08f-c6b2a5d863e2)\"" pod="llm/facebook-opt-125m-simulated-kserve-64dfc7b8f-kvtzd" podUID="755c8f99-03be-4c2f-a08f-c6b2a5d863e2" Sep 09 23:27:43.253165 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.253106 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-847796847b-rd9zd_8312264f-5e72-43fa-babf-d88e31edcccc/manager/0.log" Sep 09 23:27:43.267787 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.267760 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h2lnk/perf-node-gather-daemonset-jhxr2" Sep 09 23:27:43.273994 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.273971 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-n2htd_e9174d58-d8ff-455a-b0a1-e1df8cbd8f0d/server/0.log" Sep 09 23:27:43.341094 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.341054 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kcs88_2b53de38-ad45-40a1-956d-4bfedee7339c/manager/0.log" Sep 09 23:27:43.481656 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.481621 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68fd95c47c-zss8z_c4365b36-0d59-4e67-9c77-311cec6242fd/manager/0.log" Sep 09 23:27:43.500566 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.500536 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68fd95c47c-zss8z_c4365b36-0d59-4e67-9c77-311cec6242fd/copy-manifests/0.log" Sep 09 23:27:43.544556 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:43.544530 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:27:43.544704 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:43.544690 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:27:46.372645 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:46.372613 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6_39309e3d-3900-4ecd-9bd9-dfab92e05360/istio-proxy/0.log" Sep 09 23:27:46.437888 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:46.437841 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fj64c_8d4da4b1-7550-42cd-a17e-e1390c57beba/discovery/0.log" Sep 09 23:27:46.461285 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:46.461253 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576f68f9bc-wsjmn_07f35310-390a-48fd-b29f-9c1d08b3dd81/kube-auth-proxy/0.log" Sep 09 23:27:46.485256 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:46.485228 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/1.log" Sep 09 23:27:46.485898 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:46.485875 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/0.log" Sep 09 23:27:47.765725 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:47.765690 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-f4rw6_39309e3d-3900-4ecd-9bd9-dfab92e05360/istio-proxy/0.log" Sep 09 23:27:47.823195 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:47.823165 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fj64c_8d4da4b1-7550-42cd-a17e-e1390c57beba/discovery/0.log" Sep 09 23:27:47.854955 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:47.854923 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576f68f9bc-wsjmn_07f35310-390a-48fd-b29f-9c1d08b3dd81/kube-auth-proxy/0.log" Sep 09 23:27:47.874553 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:47.874439 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/1.log" Sep 09 23:27:47.874957 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:47.874936 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nhpdc_751437d2-bef6-4c18-b471-6514a7dcb80b/istio-proxy/0.log" Sep 09 23:27:51.541276 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.541245 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/kube-multus-additional-cni-plugins/0.log" Sep 09 23:27:51.562087 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.562058 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/egress-router-binary-copy/0.log" Sep 09 23:27:51.582981 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.582951 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/cni-plugins/0.log" Sep 09 23:27:51.602812 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.602786 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/bond-cni-plugin/0.log" Sep 09 23:27:51.623382 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.623355 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/routeoverride-cni/0.log" Sep 09 23:27:51.642385 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.642359 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/whereabouts-cni-bincopy/0.log" Sep 09 23:27:51.661996 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.661968 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sg94k_ecd37d40-2a05-4131-b63b-7dd542794f8a/whereabouts-cni/0.log" Sep 09 23:27:51.755746 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.755699 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q9z2x_81ca7c58-f6cf-45d2-892c-fbb04006ca41/kube-multus/0.log" Sep 09 23:27:51.883231 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.883131 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xbcm2_8647901c-c88a-4922-a37b-ecff32c52f6f/network-metrics-daemon/0.log" Sep 09 23:27:51.903666 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:51.903639 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xbcm2_8647901c-c88a-4922-a37b-ecff32c52f6f/kube-rbac-proxy/0.log" Sep 09 23:27:52.669516 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.669478 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/ovn-controller/0.log" Sep 09 23:27:52.700627 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.700596 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/ovn-acl-logging/0.log" Sep 09 23:27:52.739899 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.739871 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/kube-rbac-proxy-node/0.log" Sep 09 23:27:52.780031 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.780001 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 23:27:52.808272 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.808239 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/northd/0.log" Sep 09 23:27:52.837764 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.837733 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/nbdb/0.log" Sep 09 23:27:52.875853 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.875820 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/sbdb/0.log" Sep 09 23:27:52.947160 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:52.947068 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-km998_855335ea-12b5-4cfd-b507-eafb20d3e93e/ovnkube-controller/0.log" Sep 09 23:27:54.544352 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:54.544318 2571 scope.go:117] "RemoveContainer" containerID="3b8b471f550c190071d12a622f7467b2e041ffff3a59c6bd522d8fff53d55609" Sep 09 23:27:54.544733 ip-10-0-140-142 kubenswrapper[2571]: E0909 23:27:54.544547 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"main\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=main pod=e2e-distinct-simulated-kserve-6685d86975-8lqv5_llm(d30e8893-7115-40df-9db0-d2722d5f81ec)\"" pod="llm/e2e-distinct-simulated-kserve-6685d86975-8lqv5" podUID="d30e8893-7115-40df-9db0-d2722d5f81ec" Sep 09 23:27:54.759033 ip-10-0-140-142 kubenswrapper[2571]: I0909 23:27:54.758997 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-s2tnq_e8364707-07d3-4a61-bcfc-f9af03c7ed0c/network-check-target-container/0.log"