Jul 16 21:48:02.023093 ip-10-0-128-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 21:48:02.023109 ip-10-0-128-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 21:48:02.023120 ip-10-0-128-250 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 21:48:02.023463 ip-10-0-128-250 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 21:48:12.177096 ip-10-0-128-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 21:48:12.177111 ip-10-0-128-250 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 12a13bb739be49b299fb9f6edf4c2fcb -- Jul 16 21:49:46.041014 ip-10-0-128-250 systemd[1]: Starting Kubernetes Kubelet... Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.496035 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500743 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500754 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500758 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500761 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500764 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500767 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500770 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:46.552074 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500773 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500776 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500779 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500781 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500784 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500789 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500792 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500795 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500798 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500801 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500803 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500811 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500814 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500817 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500819 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500822 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500825 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500827 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500830 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:46.554099 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500833 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500836 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500838 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500841 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500844 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500846 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500849 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500851 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500854 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500857 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500860 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500862 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500865 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500868 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500870 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500873 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500875 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500878 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500880 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500883 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:46.556264 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500885 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500887 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500890 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500893 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500896 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500898 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500901 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500903 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500907 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500910 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500912 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500915 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500917 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500920 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500922 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500925 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500928 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500931 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500933 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:46.558086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500936 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500940 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500943 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500947 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500949 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500952 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500954 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500957 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500960 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500962 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500964 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500967 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500970 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500973 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500975 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500977 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500980 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500983 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500987 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:46.576349 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500990 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500992 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500996 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.500999 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501001 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501004 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501007 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501010 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501013 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501015 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501018 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501022 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501025 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501028 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501031 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501732 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501738 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501741 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501745 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501747 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:46.578388 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501751 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:46.570494 ip-10-0-128-250 systemd[1]: Started Kubernetes Kubelet. Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501755 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501759 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501762 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501766 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501769 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501772 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501775 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501778 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501780 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501783 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501786 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501789 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501792 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501795 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501797 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501800 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501803 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501806 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:46.592076 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501809 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501811 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501814 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501817 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501819 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501822 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501824 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501827 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501829 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501832 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501834 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501837 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501840 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501842 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501845 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501848 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501850 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501852 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501855 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501857 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:46.608052 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501860 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501863 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501865 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501869 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501872 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501875 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501879 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501881 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501884 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501886 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501889 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501891 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501894 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501897 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501899 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501902 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501904 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501907 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501909 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501912 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:46.610564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501917 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501919 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501922 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501924 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501928 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501930 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501932 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501935 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501938 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501941 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501943 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501945 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501948 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501950 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501953 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501956 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501958 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501961 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501964 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501967 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:46.623966 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501969 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501972 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501974 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501977 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501979 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501982 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501985 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501987 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501991 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501995 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.501998 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502001 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502005 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502008 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502011 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502356 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502361 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502364 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502367 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:46.627220 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502370 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502373 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502375 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502378 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502381 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502383 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502386 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502388 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502391 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502393 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502396 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502399 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502402 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502404 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502407 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502410 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502413 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502416 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502418 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502421 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:46.780235 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502423 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502426 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502428 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502431 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502433 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502438 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502440 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502443 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502445 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502448 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502450 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502453 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502458 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502460 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502463 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502465 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502468 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502470 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502473 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502475 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:46.801011 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502478 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502480 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502483 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502485 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502490 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502493 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502496 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502499 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502501 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502504 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502506 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502509 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502511 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502514 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502517 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502520 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502522 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502525 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502528 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:46.803269 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502530 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502533 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502535 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502538 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502540 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502544 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502546 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502549 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502552 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502554 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502557 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502559 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502562 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502564 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502567 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502569 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502571 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502574 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502577 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502579 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:46.805595 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502582 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502584 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502587 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502590 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502594 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502597 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502600 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502602 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502605 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502607 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502611 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502613 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502616 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502618 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502621 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.502623 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502691 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502699 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502704 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502721 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502725 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 21:49:46.806416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502728 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502732 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502738 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502742 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502745 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502748 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502751 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502754 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502757 2576 flags.go:64] FLAG: --cgroup-root="" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502760 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502764 2576 flags.go:64] FLAG: --client-ca-file="" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502767 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502770 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502774 2576 flags.go:64] FLAG: --cluster-domain="" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502776 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502779 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502782 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502786 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502789 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502792 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502795 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502800 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502803 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502806 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 21:49:46.808618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502809 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502812 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502815 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502818 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502822 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502825 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502828 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502831 2576 flags.go:64] FLAG: --enable-server="true" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502834 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502838 2576 flags.go:64] FLAG: --event-burst="100" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502841 2576 flags.go:64] FLAG: --event-qps="50" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502844 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502847 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502850 2576 flags.go:64] FLAG: --eviction-hard="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502853 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502856 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502859 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502862 2576 flags.go:64] FLAG: --eviction-soft="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502865 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502868 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502871 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502874 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502876 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502879 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502882 2576 flags.go:64] FLAG: --feature-gates="" Jul 16 21:49:46.810059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502885 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502888 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502891 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502894 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502898 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502901 2576 flags.go:64] FLAG: --help="false" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502904 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-250.ec2.internal" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502907 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502909 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502912 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502915 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502920 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502923 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502926 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502929 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502938 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502942 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502945 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502948 2576 flags.go:64] FLAG: --kube-reserved="" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502951 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502954 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502956 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502959 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502962 2576 flags.go:64] FLAG: --lock-file="" Jul 16 21:49:46.811224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502965 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502967 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502970 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502976 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502979 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502982 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502985 2576 flags.go:64] FLAG: --logging-format="text" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502987 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502990 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502993 2576 flags.go:64] FLAG: --manifest-url="" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.502995 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503000 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503004 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503007 2576 flags.go:64] FLAG: --max-pods="110" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503010 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503013 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503016 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503019 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503022 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503026 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503029 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503035 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503038 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503041 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 21:49:46.814194 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503044 2576 flags.go:64] FLAG: --pod-cidr="" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503047 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503052 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503054 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503057 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503060 2576 flags.go:64] FLAG: --port="10250" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503063 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503066 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08a6d7fea84e4a638" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503069 2576 flags.go:64] FLAG: --qos-reserved="" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503072 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503075 2576 flags.go:64] FLAG: --register-node="true" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503078 2576 flags.go:64] FLAG: --register-with-taints="" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503082 2576 flags.go:64] FLAG: --registry-burst="10" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503085 2576 flags.go:64] FLAG: --registry-qps="5" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503088 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503090 2576 flags.go:64] FLAG: --reserved-memory="" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503094 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503097 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503100 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503103 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503107 2576 flags.go:64] FLAG: --runonce="false" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503110 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503113 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503116 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503118 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 21:49:46.815039 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503122 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503125 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503128 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503132 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503135 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503138 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503140 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503143 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503146 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503149 2576 flags.go:64] FLAG: --system-cgroups="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503151 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503157 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503159 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503162 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503167 2576 flags.go:64] FLAG: --tls-min-version="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503169 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503172 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503175 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503178 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503181 2576 flags.go:64] FLAG: --v="2" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503185 2576 flags.go:64] FLAG: --version="false" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503189 2576 flags.go:64] FLAG: --vmodule="" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503193 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.503196 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 21:49:46.819818 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503304 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503308 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503311 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503315 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503318 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503321 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503323 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503326 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503328 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503331 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503333 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503337 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503340 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503342 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503345 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503347 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503350 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503352 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503355 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503357 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:46.820552 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503360 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503363 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503365 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503368 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503370 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503373 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503376 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503378 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503381 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503384 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503386 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503389 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503392 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503394 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503396 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503400 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503403 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503405 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503408 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:46.821126 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503412 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503415 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503417 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503420 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503426 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503429 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503432 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503435 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503438 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503440 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503443 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503445 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503448 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503450 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503453 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503456 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503458 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503460 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503463 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503465 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:46.821795 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503468 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503470 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503473 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503476 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503478 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503482 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503485 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503488 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503492 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503495 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503498 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503501 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503504 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503506 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503509 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503512 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503515 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503518 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503520 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:46.822375 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503523 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503526 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503528 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503531 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503533 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503536 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503538 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503541 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503543 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503545 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503548 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503550 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503553 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503555 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503558 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503560 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503563 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503566 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503568 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503570 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:46.822960 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.503573 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.504350 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.512987 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.513001 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.513020 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.513028 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513083 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513088 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513091 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513094 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513097 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513100 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513103 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513107 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:46.948591 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513111 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513114 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513118 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513120 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513123 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513126 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513129 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513131 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513134 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513137 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513140 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513143 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513145 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513148 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513151 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513154 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513156 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513159 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513161 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513164 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:46.949311 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513167 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513169 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513172 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513177 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513180 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513183 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513185 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513188 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513191 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513193 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513196 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513198 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513201 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513203 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513206 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513208 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513211 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513214 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513216 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513218 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:46.950117 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513221 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513226 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513229 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513232 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513234 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513238 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513242 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513244 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513247 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513250 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513252 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513255 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513257 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513260 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513262 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513267 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513269 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513271 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513274 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:46.950738 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513277 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513280 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513282 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513285 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513287 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513290 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513293 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513295 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513297 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513300 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513302 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513305 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513308 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513311 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513315 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513317 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513320 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513323 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513326 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513328 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:46.951344 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513331 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513333 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513336 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513338 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513341 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513344 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513346 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513349 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513353 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513355 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513359 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513363 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.513368 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513476 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 21:49:46.951959 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513481 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513484 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513487 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513490 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513492 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513495 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513498 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513501 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513504 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513506 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513510 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513513 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513516 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513520 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513523 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513526 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513528 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513531 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513533 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 21:49:46.952360 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513536 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513538 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513541 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513543 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513546 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513549 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513551 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513555 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513558 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513560 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513563 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513565 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513568 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513570 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513573 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513575 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513577 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513580 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513582 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513585 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 21:49:46.952918 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513587 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513590 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513592 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513595 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513598 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513603 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513607 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513610 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513613 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513615 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513618 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513620 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513623 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513625 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513628 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513630 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513633 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513635 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513638 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513642 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 21:49:46.953439 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513644 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513647 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513650 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513652 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513655 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513657 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513660 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513662 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513664 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513667 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513670 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513672 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513675 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513677 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513680 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513684 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513687 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513690 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513695 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 21:49:46.954072 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513697 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513700 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513703 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513706 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513728 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513731 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513733 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513736 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513739 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513741 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513743 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513747 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513752 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513755 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513758 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513761 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513764 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513767 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513769 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 21:49:46.954560 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:46.513772 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.513777 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.513881 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.516293 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.517946 2576 server.go:1019] "Starting client certificate rotation" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.518046 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.518076 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.520981 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.524739 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.531889 2576 log.go:25] "Validated CRI v1 runtime API" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.534340 2576 log.go:25] "Validated CRI v1 image API" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.538376 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 21:49:46.955168 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.540852 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ca790a91-3a19-4d92-a2a9-95ccd8c2c680:/dev/nvme0n1p3 e49a6fae-3351-44dc-9232-d597115f958c:/dev/nvme0n1p4] Jul 16 21:49:46.955516 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.540871 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.546778 2576 manager.go:216] Machine: {Timestamp:2026-07-16 21:49:46.544459789 +0000 UTC m=+0.365251899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102904 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2297635e5573e543fd518ae8d1dbd9 SystemUUID:ec229763-5e55-73e5-43fd-518ae8d1dbd9 BootID:12a13bb7-39be-49b2-99fb-9f6edf4c2fcb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e4:ff:a7:75:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e4:ff:a7:75:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:f7:b7:49:e8:74 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.546878 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.546960 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.549054 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.549073 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.549906 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.549915 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 21:49:46.955625 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.549926 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.549940 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.551846 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.556415 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.556932 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557034 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557058 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557068 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557078 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557087 2576 kubelet.go:398] "Adding apiserver pod source" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557095 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557429 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.557845 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.561490 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.563994 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.564023 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566175 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566188 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566194 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566199 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566208 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566213 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566221 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566226 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566233 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566239 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566247 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.566252 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.567065 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.567073 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.569348 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.569398 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.569410 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.569827 2576 server.go:1268] "Started kubelet" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.569904 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.570728 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.570784 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.570815 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.573346 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 21:49:46.956196 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.574768 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.581297 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-250.ec2.internal.18c2e38a8b09bfa7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-250.ec2.internal,UID:ip-10-0-128-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-250.ec2.internal,},FirstTimestamp:2026-07-16 21:49:46.569801639 +0000 UTC m=+0.390593749,LastTimestamp:2026-07-16 21:49:46.569801639 +0000 UTC m=+0.390593749,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-250.ec2.internal,}" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.583486 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.583498 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.583578 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584217 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584233 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584217 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584329 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584336 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584413 2576 factory.go:55] Registering systemd factory Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.584462 2576 factory.go:223] Registration of the systemd container factory successfully Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.587725 2576 factory.go:153] Registering CRI-O factory Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.587739 2576 factory.go:223] Registration of the crio container factory successfully Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.587786 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.587815 2576 factory.go:103] Registering Raw factory Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.587828 2576 manager.go:1196] Started watching for new ooms in manager Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.588217 2576 manager.go:349] Starting recovery of all containers Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.591781 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.596962 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.597061 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.598843 2576 manager.go:354] Recovery completed Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.604504 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606172 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606203 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606216 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606727 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606741 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606760 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.606729 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.607421 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.607445 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.607459 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.609407 2576 policy_none.go:49] "None policy: Start" Jul 16 21:49:46.957302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.609421 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.609433 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.611355 2576 policy_none.go:47] "Start" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.617030 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbjkx" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.617527 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-250.ec2.internal.18c2e38a8d34ec65 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-250.ec2.internal,UID:ip-10-0-128-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-250.ec2.internal,},FirstTimestamp:2026-07-16 21:49:46.606185573 +0000 UTC m=+0.426977685,LastTimestamp:2026-07-16 21:49:46.606185573 +0000 UTC m=+0.426977685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-250.ec2.internal,}" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.627298 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nbjkx" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.627379 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-250.ec2.internal.18c2e38a8d354bb9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-250.ec2.internal,UID:ip-10-0-128-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-250.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-250.ec2.internal,},FirstTimestamp:2026-07-16 21:49:46.606209977 +0000 UTC m=+0.427002091,LastTimestamp:2026-07-16 21:49:46.606209977 +0000 UTC m=+0.427002091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-250.ec2.internal,}" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.634877 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-250.ec2.internal.18c2e38a8d35751f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-250.ec2.internal,UID:ip-10-0-128-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-250.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-250.ec2.internal,},FirstTimestamp:2026-07-16 21:49:46.606220575 +0000 UTC m=+0.427012685,LastTimestamp:2026-07-16 21:49:46.606220575 +0000 UTC m=+0.427012685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-250.ec2.internal,}" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654237 2576 manager.go:337] "Starting Device Plugin manager" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.654260 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654269 2576 server.go:87] "Starting device plugin registration server" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654473 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654486 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654582 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654667 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.654678 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.655049 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.655086 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.719594 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.719630 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 21:49:47.082745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.719650 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.719682 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.722220 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.755122 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.755942 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.755962 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.755972 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.755996 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.765703 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.765730 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-250.ec2.internal\": node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.798563 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.820478 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal"] Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.820539 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.821922 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.821950 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.821960 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.824275 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.824392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.824418 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.824966 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.824985 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.824994 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.825012 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.825029 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.825039 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.827374 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.827399 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.829652 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.829676 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.829686 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.871508 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-250.ec2.internal\" not found" node="ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.083514 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.876034 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-250.ec2.internal\" not found" node="ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.886369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6750ee95623d52ab8fe01710cc128243-config\") pod \"kube-apiserver-proxy-ip-10-0-128-250.ec2.internal\" (UID: \"6750ee95623d52ab8fe01710cc128243\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.886390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/956761152313f68e26ff731b652b57b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal\" (UID: \"956761152313f68e26ff731b652b57b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.886408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/956761152313f68e26ff731b652b57b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal\" (UID: \"956761152313f68e26ff731b652b57b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:46.899282 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.987025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/956761152313f68e26ff731b652b57b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal\" (UID: \"956761152313f68e26ff731b652b57b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.987054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/956761152313f68e26ff731b652b57b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal\" (UID: \"956761152313f68e26ff731b652b57b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.987070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6750ee95623d52ab8fe01710cc128243-config\") pod \"kube-apiserver-proxy-ip-10-0-128-250.ec2.internal\" (UID: \"6750ee95623d52ab8fe01710cc128243\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.987097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6750ee95623d52ab8fe01710cc128243-config\") pod \"kube-apiserver-proxy-ip-10-0-128-250.ec2.internal\" (UID: \"6750ee95623d52ab8fe01710cc128243\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.987117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/956761152313f68e26ff731b652b57b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal\" (UID: \"956761152313f68e26ff731b652b57b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:46.987127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/956761152313f68e26ff731b652b57b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal\" (UID: \"956761152313f68e26ff731b652b57b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.084536 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.000099 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.159366 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.100884 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.225943 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:47.175076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.225943 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:47.178821 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" Jul 16 21:49:47.225943 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.201372 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.317343 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.301892 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.415672 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.402429 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.546016 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.502865 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:47.546016 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:47.517011 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 21:49:47.546016 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:47.517120 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 21:49:49.607539 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:47.603257 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:49.607539 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.593624 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:44:48 +0000 UTC" deadline="2027-12-19 18:53:11.127138679 +0000 UTC" Jul 16 21:49:49.607539 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.593650 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12501h3m21.533493562s" Jul 16 21:49:49.699514 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.624761 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:49:49.699514 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:49.663077 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:49.772091 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:49.763833 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:49.772091 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.764458 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:49:49.839196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.775123 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 21:49:49.839196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.776805 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientMemory" Jul 16 21:49:49.839196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.776834 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 21:49:49.839196 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.776843 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeHasSufficientPID" Jul 16 21:49:49.934046 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:49.864564 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:50.033878 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:49.959397 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6750ee95623d52ab8fe01710cc128243.slice/crio-b20ddbdc34a94ff82d1794fa5abb69880c8acc641f274a84dddac98f72c8d091 WatchSource:0}: Error finding container b20ddbdc34a94ff82d1794fa5abb69880c8acc641f274a84dddac98f72c8d091: Status 404 returned error can't find the container with id b20ddbdc34a94ff82d1794fa5abb69880c8acc641f274a84dddac98f72c8d091 Jul 16 21:49:50.033878 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.963571 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:49:50.033878 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:49.965344 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-250.ec2.internal\" not found" Jul 16 21:49:50.033878 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:49.969104 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:49:50.120815 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.052478 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" Jul 16 21:49:50.120815 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:50.063087 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956761152313f68e26ff731b652b57b6.slice/crio-7b2a175aecb92993486d02e211bfe707dcf74fb3e1e14f322d2f81afd3c8e731 WatchSource:0}: Error finding container 7b2a175aecb92993486d02e211bfe707dcf74fb3e1e14f322d2f81afd3c8e731: Status 404 returned error can't find the container with id 7b2a175aecb92993486d02e211bfe707dcf74fb3e1e14f322d2f81afd3c8e731 Jul 16 21:49:50.120815 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.064971 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:49:50.120815 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.066696 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" Jul 16 21:49:50.120815 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.094252 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 21:49:50.517382 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.517296 2576 apiserver.go:52] "Watching apiserver" Jul 16 21:49:50.526647 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.526621 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 21:49:50.527362 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.527329 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82","openshift-dns/node-resolver-dtt29","openshift-image-registry/node-ca-lt998","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal","openshift-multus/multus-additional-cni-plugins-s9v8c","openshift-multus/network-metrics-daemon-27sgl","kube-system/konnectivity-agent-dv7l5","openshift-cluster-node-tuning-operator/tuned-ngppg","openshift-multus/multus-t9r2v","openshift-network-diagnostics/network-check-target-lc22p","openshift-network-operator/iptables-alerter-wb68f","openshift-ovn-kubernetes/ovnkube-node-564jb","kube-system/global-pull-secret-syncer-qjjxj"] Jul 16 21:49:50.529884 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.529844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.532189 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.532164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.533134 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.533114 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.533134 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.533128 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvrrk\"" Jul 16 21:49:50.533643 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.533624 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.533723 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.533705 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 21:49:50.533807 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.533791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 21:49:50.534070 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.534058 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 21:49:50.534761 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.534551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.535510 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.535165 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-958rh\"" Jul 16 21:49:50.535510 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.535195 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 21:49:50.535510 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.535366 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 21:49:50.535510 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.535377 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.535510 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.535386 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.536612 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.536575 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 21:49:50.536612 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.536604 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 21:49:50.537009 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.536987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.538258 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.538237 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vcdm4\"" Jul 16 21:49:50.538592 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.538573 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.538669 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.538597 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.539746 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.539516 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:50.539746 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.539578 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:49:50.541523 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.541503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.542346 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.542326 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 21:49:50.542743 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.542722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 21:49:50.542743 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.542737 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l262l\"" Jul 16 21:49:50.543770 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.543758 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 21:49:50.544224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.544202 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.544773 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.544696 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 21:49:50.545293 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.545273 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 21:49:50.545378 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.545365 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2x99\"" Jul 16 21:49:50.546270 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.546251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:50.546348 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.546325 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:49:50.546415 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.546391 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p24gz\"" Jul 16 21:49:50.546547 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.546523 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.546837 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.546818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.548617 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.548601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.550981 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.550963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.551746 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.551726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 21:49:50.552541 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.552495 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.553346 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.553009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6d4mc\"" Jul 16 21:49:50.553346 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.553097 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.555253 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-log-socket\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.555312 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-cni-bin\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.555312 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-system-cni-dir\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.555416 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:50.555616 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc099ed7-cc93-4665-9d00-35bc13b47a3c-tmp-dir\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.555686 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.555686 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-cnibin\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.555779 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-cni-bin\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.555779 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a204ea0e-9533-4df2-8b0d-749f65a592ea-tmp\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.555875 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-kubelet\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.555875 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555808 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7mtkd\"" Jul 16 21:49:50.555875 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-systemd-units\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.556051 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.555891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.556051 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.556451 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556424 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.556451 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-daemon-config\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.556572 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-node-log\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.556572 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.556675 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-modprobe-d\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.556675 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556590 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 21:49:50.556768 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-run\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.556819 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovnkube-config\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.556819 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-ovn\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.556913 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cnibin\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.556913 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-tuned\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.556913 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-system-cni-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557068 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.556966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-cni-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557068 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-cni-binary-copy\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557170 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-k8s-cni-cncf-io\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557170 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557151 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-netns\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557291 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-var-lib-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.557399 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a812acaa-a375-486d-a551-d7e2e7c8b391-agent-certs\") pod \"konnectivity-agent-dv7l5\" (UID: \"a812acaa-a375-486d-a551-d7e2e7c8b391\") " pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.557399 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysctl-d\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.557486 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-kubelet\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.557486 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:50.557486 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-etc-kubernetes\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557619 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-etc-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.557619 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557525 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-run-ovn-kubernetes\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.557619 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-conf-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557756 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-slash\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.557756 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxgnh\" (UniqueName: \"kubernetes.io/projected/25591d72-2bba-45c8-b70e-6c9b0eaf4142-kube-api-access-bxgnh\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.557756 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-kubernetes\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.557893 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-host\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.557893 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475hj\" (UniqueName: \"kubernetes.io/projected/bc099ed7-cc93-4665-9d00-35bc13b47a3c-kube-api-access-475hj\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.557893 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-socket-dir-parent\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.557893 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-systemd\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.558065 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-cni-netd\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.558065 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.557956 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-systemd\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.558405 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-cni-multus\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.558487 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.558487 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-os-release\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.558487 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sk8t\" (UniqueName: \"kubernetes.io/projected/3a46827d-b431-4e78-90d1-f0cdf47dff3c-kube-api-access-4sk8t\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysconfig\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-var-lib-kubelet\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-hostroot\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-multus-certs\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.558605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:49:50.558642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558637 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovn-node-metrics-cert\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558663 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558669 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-sys\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558699 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-lib-modules\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cvs\" (UniqueName: \"kubernetes.io/projected/a204ea0e-9533-4df2-8b0d-749f65a592ea-kube-api-access-n9cvs\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc099ed7-cc93-4665-9d00-35bc13b47a3c-hosts-file\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558767 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5sqs\"" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsd67\" (UniqueName: \"kubernetes.io/projected/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-kube-api-access-xsd67\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-run-netns\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-env-overrides\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-os-release\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a812acaa-a375-486d-a551-d7e2e7c8b391-konnectivity-ca\") pod \"konnectivity-agent-dv7l5\" (UID: \"a812acaa-a375-486d-a551-d7e2e7c8b391\") " pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.558981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysctl-conf\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.559087 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.559004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7svk\" (UniqueName: \"kubernetes.io/projected/5914b657-8a03-4f03-90d5-b94f1bb5301a-kube-api-access-s7svk\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:50.559803 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.559027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovnkube-script-lib\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.559803 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.559076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 21:49:50.559803 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.559140 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 21:49:50.559803 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.559160 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 21:49:50.587462 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.587443 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tqg82" Jul 16 21:49:50.597977 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.597957 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tqg82" Jul 16 21:49:50.644400 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.644380 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 21:49:50.659801 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.659780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-os-release\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.659920 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.659818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a812acaa-a375-486d-a551-d7e2e7c8b391-konnectivity-ca\") pod \"konnectivity-agent-dv7l5\" (UID: \"a812acaa-a375-486d-a551-d7e2e7c8b391\") " pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.659920 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.659837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.659920 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.659853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysctl-conf\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.659920 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.659876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7svk\" (UniqueName: \"kubernetes.io/projected/5914b657-8a03-4f03-90d5-b94f1bb5301a-kube-api-access-s7svk\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:50.660152 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.659969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-os-release\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.660152 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysctl-conf\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.660246 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovnkube-script-lib\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.660246 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-log-socket\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.660246 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660223 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-cni-bin\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.660380 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660248 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-system-cni-dir\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.660380 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660277 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-log-socket\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.660380 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:50.660380 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc099ed7-cc93-4665-9d00-35bc13b47a3c-tmp-dir\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.660380 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660349 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-system-cni-dir\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.660380 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-socket-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-device-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-cni-bin\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-cnibin\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-cni-bin\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a204ea0e-9533-4df2-8b0d-749f65a592ea-tmp\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2897\" (UniqueName: \"kubernetes.io/projected/e0f1efb4-26b4-4616-9bb1-298b556c2d59-kube-api-access-p2897\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a812acaa-a375-486d-a551-d7e2e7c8b391-konnectivity-ca\") pod \"konnectivity-agent-dv7l5\" (UID: \"a812acaa-a375-486d-a551-d7e2e7c8b391\") " pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-cni-bin\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-cnibin\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f051cc1e-0917-46a4-bf69-e69da8dd4b65-dbus\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-kubelet\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-systemd-units\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.660671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660651 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-kubelet\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-daemon-config\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660676 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-systemd-units\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc099ed7-cc93-4665-9d00-35bc13b47a3c-tmp-dir\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-node-log\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-node-log\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-modprobe-d\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-run\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lq8\" (UniqueName: \"kubernetes.io/projected/e4d190f3-cbee-4efd-b727-fd11a101fc52-kube-api-access-q5lq8\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660862 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660912 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-modprobe-d\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.661357 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovnkube-config\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-run\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-ovn\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cnibin\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-ovn\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661213 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cnibin\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.660904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovnkube-script-lib\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661571 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-tuned\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-system-cni-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovnkube-config\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-cni-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-cni-binary-copy\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661736 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-system-cni-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-k8s-cni-cncf-io\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-netns\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-var-lib-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-daemon-config\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a812acaa-a375-486d-a551-d7e2e7c8b391-agent-certs\") pod \"konnectivity-agent-dv7l5\" (UID: \"a812acaa-a375-486d-a551-d7e2e7c8b391\") " pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysctl-d\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-netns\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-cni-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-k8s-cni-cncf-io\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-kubelet\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661893 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-var-lib-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661968 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-kubelet\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.661997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysctl-d\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.662082 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:50.662749 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-sys-fs\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-cni-binary-copy\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.662230 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:49:51.162177668 +0000 UTC m=+3.023752429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-etc-kubernetes\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-etc-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-run-ovn-kubernetes\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-registration-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662364 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-etc-kubernetes\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-iptables-alerter-script\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-conf-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-run-ovn-kubernetes\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-etc-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.663542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-slash\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxgnh\" (UniqueName: \"kubernetes.io/projected/25591d72-2bba-45c8-b70e-6c9b0eaf4142-kube-api-access-bxgnh\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-slash\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-conf-dir\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-kubernetes\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-host\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475hj\" (UniqueName: \"kubernetes.io/projected/bc099ed7-cc93-4665-9d00-35bc13b47a3c-kube-api-access-475hj\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-kubernetes\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-host-slash\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662597 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-host\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-socket-dir-parent\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-systemd\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-cni-netd\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662781 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-multus-socket-dir-parent\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-systemd\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664208 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-systemd\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwb5\" (UniqueName: \"kubernetes.io/projected/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-kube-api-access-5nwb5\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-cni-netd\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-systemd\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4d190f3-cbee-4efd-b727-fd11a101fc52-host\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-cni-multus\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-os-release\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.662975 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-var-lib-cni-multus\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-run-openvswitch\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sk8t\" (UniqueName: \"kubernetes.io/projected/3a46827d-b431-4e78-90d1-f0cdf47dff3c-kube-api-access-4sk8t\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-os-release\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysconfig\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.664859 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-var-lib-kubelet\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4d190f3-cbee-4efd-b727-fd11a101fc52-serviceca\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663131 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-hostroot\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-multus-certs\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovn-node-metrics-cert\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663208 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-sys\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-lib-modules\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-sysconfig\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cvs\" (UniqueName: \"kubernetes.io/projected/a204ea0e-9533-4df2-8b0d-749f65a592ea-kube-api-access-n9cvs\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc099ed7-cc93-4665-9d00-35bc13b47a3c-hosts-file\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-var-lib-kubelet\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsd67\" (UniqueName: \"kubernetes.io/projected/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-kube-api-access-xsd67\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663345 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-run-netns\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-env-overrides\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-sys\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.665568 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663427 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-hostroot\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f051cc1e-0917-46a4-bf69-e69da8dd4b65-kubelet-config\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-host-run-multus-certs\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bc099ed7-cc93-4665-9d00-35bc13b47a3c-hosts-file\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a204ea0e-9533-4df2-8b0d-749f65a592ea-lib-modules\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663946 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a46827d-b431-4e78-90d1-f0cdf47dff3c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/25591d72-2bba-45c8-b70e-6c9b0eaf4142-host-run-netns\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.663990 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/25591d72-2bba-45c8-b70e-6c9b0eaf4142-env-overrides\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.664161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3a46827d-b431-4e78-90d1-f0cdf47dff3c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.664717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a204ea0e-9533-4df2-8b0d-749f65a592ea-tmp\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.664763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a204ea0e-9533-4df2-8b0d-749f65a592ea-etc-tuned\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.665665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/25591d72-2bba-45c8-b70e-6c9b0eaf4142-ovn-node-metrics-cert\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.666340 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.666092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a812acaa-a375-486d-a551-d7e2e7c8b391-agent-certs\") pod \"konnectivity-agent-dv7l5\" (UID: \"a812acaa-a375-486d-a551-d7e2e7c8b391\") " pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.669543 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.669516 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7svk\" (UniqueName: \"kubernetes.io/projected/5914b657-8a03-4f03-90d5-b94f1bb5301a-kube-api-access-s7svk\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:50.671318 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.671300 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:49:50.671441 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.671322 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:49:50.671441 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.671390 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:50.671603 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.671490 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:51.171473945 +0000 UTC m=+3.033048701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:50.673942 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.673884 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-475hj\" (UniqueName: \"kubernetes.io/projected/bc099ed7-cc93-4665-9d00-35bc13b47a3c-kube-api-access-475hj\") pod \"node-resolver-dtt29\" (UID: \"bc099ed7-cc93-4665-9d00-35bc13b47a3c\") " pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.675103 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.675062 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsd67\" (UniqueName: \"kubernetes.io/projected/5835d2d8-4b2e-47fe-813f-dd649c70d3fe-kube-api-access-xsd67\") pod \"multus-t9r2v\" (UID: \"5835d2d8-4b2e-47fe-813f-dd649c70d3fe\") " pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.675659 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.675640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sk8t\" (UniqueName: \"kubernetes.io/projected/3a46827d-b431-4e78-90d1-f0cdf47dff3c-kube-api-access-4sk8t\") pod \"multus-additional-cni-plugins-s9v8c\" (UID: \"3a46827d-b431-4e78-90d1-f0cdf47dff3c\") " pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.675882 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.675851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cvs\" (UniqueName: \"kubernetes.io/projected/a204ea0e-9533-4df2-8b0d-749f65a592ea-kube-api-access-n9cvs\") pod \"tuned-ngppg\" (UID: \"a204ea0e-9533-4df2-8b0d-749f65a592ea\") " pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.675972 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.675954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxgnh\" (UniqueName: \"kubernetes.io/projected/25591d72-2bba-45c8-b70e-6c9b0eaf4142-kube-api-access-bxgnh\") pod \"ovnkube-node-564jb\" (UID: \"25591d72-2bba-45c8-b70e-6c9b0eaf4142\") " pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.686326 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.686278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" event={"ID":"956761152313f68e26ff731b652b57b6","Type":"ContainerStarted","Data":"7b2a175aecb92993486d02e211bfe707dcf74fb3e1e14f322d2f81afd3c8e731"} Jul 16 21:49:50.689054 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.689032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" event={"ID":"6750ee95623d52ab8fe01710cc128243","Type":"ContainerStarted","Data":"b20ddbdc34a94ff82d1794fa5abb69880c8acc641f274a84dddac98f72c8d091"} Jul 16 21:49:50.764234 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lq8\" (UniqueName: \"kubernetes.io/projected/e4d190f3-cbee-4efd-b727-fd11a101fc52-kube-api-access-q5lq8\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.764234 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.764425 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764425 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764425 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-sys-fs\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764425 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-registration-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764425 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764425 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-sys-fs\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.764443 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-etc-selinux\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-iptables-alerter-script\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:50.764506 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:51.264487628 +0000 UTC m=+3.126062375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764521 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-registration-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-host-slash\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764568 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwb5\" (UniqueName: \"kubernetes.io/projected/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-kube-api-access-5nwb5\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4d190f3-cbee-4efd-b727-fd11a101fc52-host\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-host-slash\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.764646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4d190f3-cbee-4efd-b727-fd11a101fc52-serviceca\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f051cc1e-0917-46a4-bf69-e69da8dd4b65-kubelet-config\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764672 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e4d190f3-cbee-4efd-b727-fd11a101fc52-host\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764726 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f051cc1e-0917-46a4-bf69-e69da8dd4b65-kubelet-config\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-socket-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-device-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2897\" (UniqueName: \"kubernetes.io/projected/e0f1efb4-26b4-4616-9bb1-298b556c2d59-kube-api-access-p2897\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-socket-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764892 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f051cc1e-0917-46a4-bf69-e69da8dd4b65-dbus\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e0f1efb4-26b4-4616-9bb1-298b556c2d59-device-dir\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764980 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f051cc1e-0917-46a4-bf69-e69da8dd4b65-dbus\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.764982 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-iptables-alerter-script\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.765119 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.765123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e4d190f3-cbee-4efd-b727-fd11a101fc52-serviceca\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.773273 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.773245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2897\" (UniqueName: \"kubernetes.io/projected/e0f1efb4-26b4-4616-9bb1-298b556c2d59-kube-api-access-p2897\") pod \"aws-ebs-csi-driver-node-6dp82\" (UID: \"e0f1efb4-26b4-4616-9bb1-298b556c2d59\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.773464 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.773445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwb5\" (UniqueName: \"kubernetes.io/projected/d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179-kube-api-access-5nwb5\") pod \"iptables-alerter-wb68f\" (UID: \"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179\") " pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.773540 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.773523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lq8\" (UniqueName: \"kubernetes.io/projected/e4d190f3-cbee-4efd-b727-fd11a101fc52-kube-api-access-q5lq8\") pod \"node-ca-lt998\" (UID: \"e4d190f3-cbee-4efd-b727-fd11a101fc52\") " pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:50.843727 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.843658 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t9r2v" Jul 16 21:49:50.852234 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.852216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dtt29" Jul 16 21:49:50.854853 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:50.854825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5835d2d8_4b2e_47fe_813f_dd649c70d3fe.slice/crio-32c494b78e5fce20006b7716768fd0d449f2ff0d3df514bf3ec14fabb2696441 WatchSource:0}: Error finding container 32c494b78e5fce20006b7716768fd0d449f2ff0d3df514bf3ec14fabb2696441: Status 404 returned error can't find the container with id 32c494b78e5fce20006b7716768fd0d449f2ff0d3df514bf3ec14fabb2696441 Jul 16 21:49:50.861086 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:50.861065 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc099ed7_cc93_4665_9d00_35bc13b47a3c.slice/crio-1fd1e50c9df0f9359805520c4d2d0fe934041511b9291e6d0a351cbe2318690c WatchSource:0}: Error finding container 1fd1e50c9df0f9359805520c4d2d0fe934041511b9291e6d0a351cbe2318690c: Status 404 returned error can't find the container with id 1fd1e50c9df0f9359805520c4d2d0fe934041511b9291e6d0a351cbe2318690c Jul 16 21:49:50.864953 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.864937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:49:50.870011 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.869988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:49:50.877322 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.877296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" Jul 16 21:49:50.886016 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.885990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ngppg" Jul 16 21:49:50.892815 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.892794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" Jul 16 21:49:50.901532 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.901514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wb68f" Jul 16 21:49:50.906326 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:50.906310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lt998" Jul 16 21:49:51.166454 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.166364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:51.166602 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.166536 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:51.166656 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.166611 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:49:52.166592417 +0000 UTC m=+4.028167180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:51.267635 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.267588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:51.267799 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.267682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:51.267799 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.267722 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:51.267799 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.267793 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:49:51.267960 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.267802 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:52.267782626 +0000 UTC m=+4.129357379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:51.267960 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.267810 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:49:51.267960 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.267819 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:51.267960 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.267859 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:52.267844246 +0000 UTC m=+4.129418997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:51.601050 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.600954 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:44:50 +0000 UTC" deadline="2028-05-02 00:39:45.701997996 +0000 UTC" Jul 16 21:49:51.601050 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.601016 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15722h49m54.100991923s" Jul 16 21:49:51.679772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.679698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:51.680319 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.679838 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:49:51.680319 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.679698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:51.680319 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:51.680285 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:49:51.692679 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.692655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtt29" event={"ID":"bc099ed7-cc93-4665-9d00-35bc13b47a3c","Type":"ContainerStarted","Data":"1fd1e50c9df0f9359805520c4d2d0fe934041511b9291e6d0a351cbe2318690c"} Jul 16 21:49:51.693947 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:51.693906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t9r2v" event={"ID":"5835d2d8-4b2e-47fe-813f-dd649c70d3fe","Type":"ContainerStarted","Data":"32c494b78e5fce20006b7716768fd0d449f2ff0d3df514bf3ec14fabb2696441"} Jul 16 21:49:51.983343 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:51.983304 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93dcc0b_98cc_4c2c_b5bc_2cf7fbe14179.slice/crio-2d6b147f7a0f3b03da577c643f91df4a047926329b29c7ca805f21d1decde193 WatchSource:0}: Error finding container 2d6b147f7a0f3b03da577c643f91df4a047926329b29c7ca805f21d1decde193: Status 404 returned error can't find the container with id 2d6b147f7a0f3b03da577c643f91df4a047926329b29c7ca805f21d1decde193 Jul 16 21:49:51.990859 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:51.990747 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda812acaa_a375_486d_a551_d7e2e7c8b391.slice/crio-3392e1ea1228129dd38b912468e4ecd1b2a900ef25d8cd1b3361e71361b66d5e WatchSource:0}: Error finding container 3392e1ea1228129dd38b912468e4ecd1b2a900ef25d8cd1b3361e71361b66d5e: Status 404 returned error can't find the container with id 3392e1ea1228129dd38b912468e4ecd1b2a900ef25d8cd1b3361e71361b66d5e Jul 16 21:49:51.996165 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:51.996088 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda204ea0e_9533_4df2_8b0d_749f65a592ea.slice/crio-ebf46598d366457171ac1be834ba778939ddf3406c68255dc1bf38b564ce47fe WatchSource:0}: Error finding container ebf46598d366457171ac1be834ba778939ddf3406c68255dc1bf38b564ce47fe: Status 404 returned error can't find the container with id ebf46598d366457171ac1be834ba778939ddf3406c68255dc1bf38b564ce47fe Jul 16 21:49:51.996464 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:49:51.996421 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a46827d_b431_4e78_90d1_f0cdf47dff3c.slice/crio-699089a9e6a6cee5e6fbfa0125c1dbf86e9b659badea59a8078c78b16f4ef08b WatchSource:0}: Error finding container 699089a9e6a6cee5e6fbfa0125c1dbf86e9b659badea59a8078c78b16f4ef08b: Status 404 returned error can't find the container with id 699089a9e6a6cee5e6fbfa0125c1dbf86e9b659badea59a8078c78b16f4ef08b Jul 16 21:49:52.174040 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.173996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:52.174241 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.174215 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:52.174310 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.174296 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:49:54.174274429 +0000 UTC m=+6.035849189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:52.274963 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.274923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:52.275091 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.274987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:52.275402 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.275360 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:52.275530 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.275514 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:54.275484082 +0000 UTC m=+6.137058837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:52.275608 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.275515 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:49:52.275608 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.275554 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:49:52.275608 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.275570 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:52.275741 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.275622 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:54.275608236 +0000 UTC m=+6.137182991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:52.602157 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.601724 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:44:50 +0000 UTC" deadline="2028-01-18 04:11:53.568058718 +0000 UTC" Jul 16 21:49:52.602157 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.602012 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13206h22m0.966052834s" Jul 16 21:49:52.682469 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.681987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:52.682469 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:52.682109 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:49:52.700135 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.700107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ngppg" event={"ID":"a204ea0e-9533-4df2-8b0d-749f65a592ea","Type":"ContainerStarted","Data":"ebf46598d366457171ac1be834ba778939ddf3406c68255dc1bf38b564ce47fe"} Jul 16 21:49:52.712694 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.712643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dv7l5" event={"ID":"a812acaa-a375-486d-a551-d7e2e7c8b391","Type":"ContainerStarted","Data":"3392e1ea1228129dd38b912468e4ecd1b2a900ef25d8cd1b3361e71361b66d5e"} Jul 16 21:49:52.715885 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.715810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wb68f" event={"ID":"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179","Type":"ContainerStarted","Data":"2d6b147f7a0f3b03da577c643f91df4a047926329b29c7ca805f21d1decde193"} Jul 16 21:49:52.724050 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.724022 2576 generic.go:356] "Generic (PLEG): container finished" podID="956761152313f68e26ff731b652b57b6" containerID="c80dbfbf0be3e1e17e849dd6f81eced3806103baaaa6feb6ed56b50545fed2f0" exitCode=0 Jul 16 21:49:52.724150 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.724110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" event={"ID":"956761152313f68e26ff731b652b57b6","Type":"ContainerDied","Data":"c80dbfbf0be3e1e17e849dd6f81eced3806103baaaa6feb6ed56b50545fed2f0"} Jul 16 21:49:52.728910 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.727901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"155d3f6a21a3ef17b4ce17ebfa95d3e6213742a44d995d30850462131319f993"} Jul 16 21:49:52.730511 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.730486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lt998" event={"ID":"e4d190f3-cbee-4efd-b727-fd11a101fc52","Type":"ContainerStarted","Data":"fca4c3c8c4b6bbcd661d4bb314c46d5bb34a6b8a086e002787983789c80c6bd9"} Jul 16 21:49:52.740738 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.740715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" event={"ID":"e0f1efb4-26b4-4616-9bb1-298b556c2d59","Type":"ContainerStarted","Data":"2c2638b7e2594fe11e6dd9deba55a17a172d6a4534ad52c4d603ec6079a5cdd6"} Jul 16 21:49:52.749310 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.749285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" event={"ID":"6750ee95623d52ab8fe01710cc128243","Type":"ContainerStarted","Data":"6f2622e976351d04702176ad6dfc40427c3d73c8abf0cf10024233efa999db84"} Jul 16 21:49:52.753414 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:52.753390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerStarted","Data":"699089a9e6a6cee5e6fbfa0125c1dbf86e9b659badea59a8078c78b16f4ef08b"} Jul 16 21:49:53.679723 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:53.679038 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:53.679723 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:53.679160 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:49:53.679723 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:53.679563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:53.679723 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:53.679664 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:49:54.190038 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:54.189994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:54.190534 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.190177 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:54.190534 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.190265 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:49:58.190245032 +0000 UTC m=+10.051819849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:54.290909 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:54.290872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:54.291107 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:54.290944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:54.291107 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.291094 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:54.291229 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.291154 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:58.291136796 +0000 UTC m=+10.152711538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:54.291580 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.291557 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:49:54.291706 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.291584 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:49:54.291706 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.291596 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:54.291706 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.291642 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:58.29162692 +0000 UTC m=+10.153201674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:54.682180 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:54.682079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:54.682367 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:54.682245 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:49:55.679740 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:55.679393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:55.679740 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:55.679416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:55.679740 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:55.679549 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:49:55.679740 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:55.679715 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:49:56.682107 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:56.682007 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:56.682582 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:56.682253 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:49:57.679515 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:57.679359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:57.679515 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:57.679466 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:49:57.679824 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:57.679574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:57.679824 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:57.679686 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:49:58.227232 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:58.227183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:58.227709 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.227353 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:58.227709 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.227417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:06.227398205 +0000 UTC m=+18.088972968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:49:58.327990 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:58.327952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:58.328156 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:58.328012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:58.328156 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.328114 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:58.328156 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.328137 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:49:58.328330 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.328163 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:49:58.328330 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.328170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:06.328152829 +0000 UTC m=+18.189727590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:49:58.328330 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.328176 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:58.328330 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.328227 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:06.32821044 +0000 UTC m=+18.189785203 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:49:58.679765 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:58.679686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:49:58.679921 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:58.679817 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:49:59.679341 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:59.679248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:49:59.679341 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:49:59.679280 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:49:59.679852 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:59.679407 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:49:59.679852 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:49:59.679788 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:00.680212 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:00.680178 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:00.680570 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:00.680316 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:01.679792 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:01.679754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:01.679792 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:01.679780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:01.680031 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:01.679868 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:01.680031 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:01.679955 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:02.679366 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:02.679331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:02.679786 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:02.679482 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:03.679509 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:03.679479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:03.679973 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:03.679608 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:03.679973 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:03.679657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:03.679973 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:03.679751 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:04.679559 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:04.679524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:04.680023 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:04.679657 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:05.679415 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:05.679372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:05.679581 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:05.679485 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:05.679581 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:05.679545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:05.679842 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:05.679635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:06.288922 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:06.288886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:06.289097 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.289035 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:06.289172 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.289106 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:22.289086851 +0000 UTC m=+34.150661606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:06.389837 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:06.389801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:06.390003 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:06.389849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:06.390003 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.389979 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:06.390198 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.389998 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:06.390198 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.390026 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:06.390198 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.390037 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:22.390020778 +0000 UTC m=+34.251595535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:06.390198 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.390039 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:06.390198 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.390079 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:22.390069678 +0000 UTC m=+34.251644431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:06.682502 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:06.682423 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:06.682884 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:06.682534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:07.679085 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:07.679051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:07.679270 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:07.679087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:07.679270 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:07.679195 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:07.679414 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:07.679302 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:08.680438 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:08.680407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:08.680965 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:08.680507 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:09.679614 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:09.679579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:09.679794 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:09.679622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:09.679794 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:09.679702 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:09.679904 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:09.679844 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:10.680157 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.680006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:10.680662 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:10.680235 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:10.795386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.795356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t9r2v" event={"ID":"5835d2d8-4b2e-47fe-813f-dd649c70d3fe","Type":"ContainerStarted","Data":"62e4ee9db23a036987ffd4d8040b0d74927312ddaf2f2744f362cea419ad03f3"} Jul 16 21:50:10.796684 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.796661 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a46827d-b431-4e78-90d1-f0cdf47dff3c" containerID="ab6a5c95ec8eb420c183d9fd7cea5544547cc47200dfe8db977b30013383debd" exitCode=0 Jul 16 21:50:10.796783 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.796724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerDied","Data":"ab6a5c95ec8eb420c183d9fd7cea5544547cc47200dfe8db977b30013383debd"} Jul 16 21:50:10.798320 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.798291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ngppg" event={"ID":"a204ea0e-9533-4df2-8b0d-749f65a592ea","Type":"ContainerStarted","Data":"86c196e75ef6a6a3ef658cfaaaa04544937055deeb6636ba41e2b87a3c1c73c2"} Jul 16 21:50:10.799677 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.799659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dv7l5" event={"ID":"a812acaa-a375-486d-a551-d7e2e7c8b391","Type":"ContainerStarted","Data":"c4837263b62332288cb58ae3b77d05dda3161ce849c55b78932597a56259ea69"} Jul 16 21:50:10.801393 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.801361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" event={"ID":"956761152313f68e26ff731b652b57b6","Type":"ContainerStarted","Data":"c876f896b87be4acb5e95be7ff6add937734f64e2c9a4d55dc6f81c0bfb8d295"} Jul 16 21:50:10.804462 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.804436 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dtt29" event={"ID":"bc099ed7-cc93-4665-9d00-35bc13b47a3c","Type":"ContainerStarted","Data":"b82fb0eeefef1f8db80883abf98a8365170a5fb5b63138de8f09dff391eeb485"} Jul 16 21:50:10.808103 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.808080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"289add1197eaa2b36cf369d2e52a34dee099dae136284d3b121289633e6907e3"} Jul 16 21:50:10.808198 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.808157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"8ec26b871d39fbea0976e1300bb00e707f7a7587aa597be4554edb715d8d41dd"} Jul 16 21:50:10.808242 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.808209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"ca7a53523fbcefa760f25d6d1af6caf258688396e91b2cdc98f38f3362c4e26f"} Jul 16 21:50:10.808273 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.808258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"e775677b24208a1fe1a0b4d6e4a125b169a7cb9c31304cf5a17234448a3cf5d0"} Jul 16 21:50:10.809421 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.809403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lt998" event={"ID":"e4d190f3-cbee-4efd-b727-fd11a101fc52","Type":"ContainerStarted","Data":"760eeab8922c9b6a1349b2f28c1e076a245e2c8ccd46b830237f1e0766ff8f71"} Jul 16 21:50:10.810698 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.810676 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" event={"ID":"e0f1efb4-26b4-4616-9bb1-298b556c2d59","Type":"ContainerStarted","Data":"8cfd878f2414cd65bfa5635df932b3c6e9d6b1a5cf75986e1082bcdc8637e7c7"} Jul 16 21:50:10.830961 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.830899 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-250.ec2.internal" podStartSLOduration=20.830883758 podStartE2EDuration="20.830883758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:52.765968837 +0000 UTC m=+4.627543599" watchObservedRunningTime="2026-07-16 21:50:10.830883758 +0000 UTC m=+22.692458518" Jul 16 21:50:10.831365 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.831336 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t9r2v" podStartSLOduration=3.456671561 podStartE2EDuration="22.831326719s" totalImagesPullingTime="19.374655158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:10.830979939 +0000 UTC m=+22.692554701" watchObservedRunningTime="2026-07-16 21:50:10.831326719 +0000 UTC m=+22.692901481" Jul 16 21:50:10.855275 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.855236 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-250.ec2.internal" podStartSLOduration=20.855226321 podStartE2EDuration="20.855226321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:10.855140831 +0000 UTC m=+22.716715592" watchObservedRunningTime="2026-07-16 21:50:10.855226321 +0000 UTC m=+22.716801083" Jul 16 21:50:10.873855 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.873819 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dv7l5" podStartSLOduration=4.638305914 podStartE2EDuration="22.873808738s" totalImagesPullingTime="18.235502824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:10.873291083 +0000 UTC m=+22.734865843" watchObservedRunningTime="2026-07-16 21:50:10.873808738 +0000 UTC m=+22.735383498" Jul 16 21:50:10.914102 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.914045 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ngppg" podStartSLOduration=4.679017174 podStartE2EDuration="22.914027511s" totalImagesPullingTime="18.235010337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:10.913633894 +0000 UTC m=+22.775208654" watchObservedRunningTime="2026-07-16 21:50:10.914027511 +0000 UTC m=+22.775602272" Jul 16 21:50:10.930437 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.930400 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dtt29" podStartSLOduration=3.986498265 podStartE2EDuration="22.930389002s" totalImagesPullingTime="18.943890737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:10.930226167 +0000 UTC m=+22.791800941" watchObservedRunningTime="2026-07-16 21:50:10.930389002 +0000 UTC m=+22.791963764" Jul 16 21:50:10.945657 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:10.945624 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lt998" podStartSLOduration=4.146586851 podStartE2EDuration="21.945616302s" totalImagesPullingTime="17.799029451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:10.945524543 +0000 UTC m=+22.807099304" watchObservedRunningTime="2026-07-16 21:50:10.945616302 +0000 UTC m=+22.807191063" Jul 16 21:50:11.679165 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:11.679138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:11.679443 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:11.679146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:11.679443 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:11.679240 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:11.679443 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:11.679328 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:11.814539 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:11.814511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wb68f" event={"ID":"d93dcc0b-98cc-4c2c-b5bc-2cf7fbe14179","Type":"ContainerStarted","Data":"ca89f9f5011e543fc4266c4f8d8105fd827345310998e29eba84508e08a0c2dc"} Jul 16 21:50:11.817180 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:11.817159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"f1830351d833908b6f0e810ef2e9de338682dd88777300d7883c4b6081c3331f"} Jul 16 21:50:11.817276 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:11.817216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"4b3de620c819cf66e118bee22608b05828da057494bad3f3812f9fb43621a18c"} Jul 16 21:50:11.832560 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:11.832522 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wb68f" podStartSLOduration=5.012677187 podStartE2EDuration="22.832512247s" totalImagesPullingTime="17.81983506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:11.832337614 +0000 UTC m=+23.693912375" watchObservedRunningTime="2026-07-16 21:50:11.832512247 +0000 UTC m=+23.694087007" Jul 16 21:50:12.089552 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:12.089528 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 21:50:12.627270 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:12.627160 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:50:12.089547905Z","UUID":"d1694d83-3fa2-4923-a4df-42b04aabdf79","Handler":null,"Name":"","Endpoint":""} Jul 16 21:50:12.629461 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:12.629434 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 21:50:12.629607 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:12.629482 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 21:50:12.683492 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:12.683465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:12.683616 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:12.683564 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:12.821887 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:12.821855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" event={"ID":"e0f1efb4-26b4-4616-9bb1-298b556c2d59","Type":"ContainerStarted","Data":"ca3da16a01bf3c1221d2f430bd3ff81afedb1914d028b4fed33675ad872a6de2"} Jul 16 21:50:13.680017 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:13.679936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:13.680282 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:13.679938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:13.680282 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:13.680060 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:13.680282 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:13.680173 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:13.827014 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:13.826965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"c54d1b78c677cdfddb5c53b8c6ab3e50e9ec9d7d115a4219211f2549946a26d5"} Jul 16 21:50:13.829382 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:13.829354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" event={"ID":"e0f1efb4-26b4-4616-9bb1-298b556c2d59","Type":"ContainerStarted","Data":"4313d06b734fa4ef1a656e1b6c8b5e09bb71fef91387b38f0a075c961874e474"} Jul 16 21:50:14.039553 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:14.039252 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:50:14.039966 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:14.039941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:50:14.060134 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:14.060090 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6dp82" podStartSLOduration=4.159590066 podStartE2EDuration="25.060074609s" totalImagesPullingTime="20.900484543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:13.851859068 +0000 UTC m=+25.713433828" watchObservedRunningTime="2026-07-16 21:50:14.060074609 +0000 UTC m=+25.921649369" Jul 16 21:50:14.682865 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:14.682828 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:14.683055 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:14.682951 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:14.832010 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:14.831857 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:50:14.832412 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:14.832329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dv7l5" Jul 16 21:50:15.679951 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.679916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:15.680078 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:15.680055 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:15.680134 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.680111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:15.680239 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:15.680220 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:15.838761 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.838563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" event={"ID":"25591d72-2bba-45c8-b70e-6c9b0eaf4142","Type":"ContainerStarted","Data":"285e15455c1a5d60c519239bbe2fc54ec92a42b7feb0708133088f7d89ff4454"} Jul 16 21:50:15.839435 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.839173 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:50:15.839435 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.839338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:50:15.839526 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.839483 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:50:15.865041 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.865017 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:50:15.865167 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.865155 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:50:15.872144 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:15.872106 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" podStartSLOduration=9.573331225 podStartE2EDuration="27.872094171s" totalImagesPullingTime="18.298762946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:15.871530461 +0000 UTC m=+27.733105222" watchObservedRunningTime="2026-07-16 21:50:15.872094171 +0000 UTC m=+27.733668932" Jul 16 21:50:16.679129 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:16.679098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:16.679302 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:16.679231 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:16.842638 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:16.842607 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a46827d-b431-4e78-90d1-f0cdf47dff3c" containerID="686715b7ed9381ed374ab306ea88fd02cd641358fef28d50a512ca9b6c3ff436" exitCode=0 Jul 16 21:50:16.843034 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:16.842689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerDied","Data":"686715b7ed9381ed374ab306ea88fd02cd641358fef28d50a512ca9b6c3ff436"} Jul 16 21:50:17.671844 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.671816 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lc22p"] Jul 16 21:50:17.672067 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.671956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:17.672145 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:17.672061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:17.675633 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.675609 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qjjxj"] Jul 16 21:50:17.675729 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.675697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:17.675795 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:17.675782 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:17.679390 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.679364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:17.679512 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:17.679492 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:17.686510 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.686483 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-27sgl"] Jul 16 21:50:17.846341 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.846315 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a46827d-b431-4e78-90d1-f0cdf47dff3c" containerID="caa7175f84f150c7da7ee4ef825953ff55f2223957375958a17fb3540983cb6a" exitCode=0 Jul 16 21:50:17.846708 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.846382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:17.846708 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:17.846395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerDied","Data":"caa7175f84f150c7da7ee4ef825953ff55f2223957375958a17fb3540983cb6a"} Jul 16 21:50:17.847010 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:17.846991 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:19.679526 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:19.679276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:19.679887 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:19.679289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:19.679887 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:19.679645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:19.679887 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:19.679291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:19.679887 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:19.679693 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:19.679887 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:19.679734 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:19.853202 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:19.853175 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a46827d-b431-4e78-90d1-f0cdf47dff3c" containerID="0d8f1b93d666ad29ddd1114f55690aa8ca8fad5ec884fdddec007ada946550e6" exitCode=0 Jul 16 21:50:19.853310 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:19.853218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerDied","Data":"0d8f1b93d666ad29ddd1114f55690aa8ca8fad5ec884fdddec007ada946550e6"} Jul 16 21:50:21.679664 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:21.679633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:21.680193 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:21.679649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:21.680193 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:21.679760 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qjjxj" podUID="f051cc1e-0917-46a4-bf69-e69da8dd4b65" Jul 16 21:50:21.680193 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:21.679782 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:21.680193 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:21.679891 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lc22p" podUID="92104d61-3706-4d0c-beb9-ce3905eb8740" Jul 16 21:50:21.680193 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:21.680030 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-27sgl" podUID="5914b657-8a03-4f03-90d5-b94f1bb5301a" Jul 16 21:50:22.313471 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:22.313439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:22.313632 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.313559 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:22.313632 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.313612 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:54.313593565 +0000 UTC m=+66.175168307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 21:50:22.413955 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:22.413905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:22.414131 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:22.413978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:22.414131 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.414086 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:22.414131 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.414090 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 21:50:22.414131 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.414114 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 21:50:22.414131 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.414126 2576 projected.go:196] Error preparing data for projected volume kube-api-access-7q8j7 for pod openshift-network-diagnostics/network-check-target-lc22p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:22.414418 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.414142 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret podName:f051cc1e-0917-46a4-bf69-e69da8dd4b65 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:54.414124726 +0000 UTC m=+66.275699465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret") pod "global-pull-secret-syncer-qjjxj" (UID: "f051cc1e-0917-46a4-bf69-e69da8dd4b65") : object "kube-system"/"original-pull-secret" not registered Jul 16 21:50:22.414418 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:22.414176 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7 podName:92104d61-3706-4d0c-beb9-ce3905eb8740 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:54.414159943 +0000 UTC m=+66.275734701 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7q8j7" (UniqueName: "kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7") pod "network-check-target-lc22p" (UID: "92104d61-3706-4d0c-beb9-ce3905eb8740") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 21:50:22.950908 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:22.950882 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-250.ec2.internal" event="NodeReady" Jul 16 21:50:22.951398 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:22.951031 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 21:50:22.997183 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:22.997154 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57749b7b4c-6wtsb"] Jul 16 21:50:23.002382 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.002361 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.005243 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.005214 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n"] Jul 16 21:50:23.008726 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.008704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj"] Jul 16 21:50:23.008854 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.008808 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.012461 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.012440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.013501 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.013430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 21:50:23.014315 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.014109 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 21:50:23.014315 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.014222 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 16 21:50:23.014897 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.014875 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29"] Jul 16 21:50:23.015047 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.015033 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nmm9q\"" Jul 16 21:50:23.015259 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.015241 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 21:50:23.015453 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.015424 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 21:50:23.015964 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.015943 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 21:50:23.016159 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.016144 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 16 21:50:23.016471 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.016450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 21:50:23.017566 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.017409 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 16 21:50:23.018352 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.018329 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 16 21:50:23.018497 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.018478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 16 21:50:23.018855 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.018819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.020822 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.020799 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 21:50:23.024486 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.024468 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-588zz"] Jul 16 21:50:23.030080 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.029047 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tw46w"] Jul 16 21:50:23.030080 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.029361 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 16 21:50:23.030080 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.029698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.030445 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.030425 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mkvdp\"" Jul 16 21:50:23.034951 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.032841 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n6rt\"" Jul 16 21:50:23.034951 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.033271 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 21:50:23.034951 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.033389 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57749b7b4c-6wtsb"] Jul 16 21:50:23.034951 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.033430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29"] Jul 16 21:50:23.034951 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.033735 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.035240 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.035156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n"] Jul 16 21:50:23.036810 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.036788 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 21:50:23.037758 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.037738 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 21:50:23.038131 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.038000 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pz5gf\"" Jul 16 21:50:23.038230 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.037740 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 21:50:23.038738 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.038717 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 21:50:23.039326 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.039306 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tw46w"] Jul 16 21:50:23.042910 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.042891 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-588zz"] Jul 16 21:50:23.052173 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.052150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj"] Jul 16 21:50:23.118845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.118810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.118845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.118845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e702f9-c995-4223-99c2-900fa238ff8a-tmp-dir\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.119066 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.118863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt5s2\" (UniqueName: \"kubernetes.io/projected/66e702f9-c995-4223-99c2-900fa238ff8a-kube-api-access-rt5s2\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.119066 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.118888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq42k\" (UniqueName: \"kubernetes.io/projected/4e869a5f-b91c-489d-b3be-851f22dbee7a-kube-api-access-wq42k\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.119066 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.118959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-ca\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.119066 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.118976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119066 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119002 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.119237 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e869a5f-b91c-489d-b3be-851f22dbee7a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.119237 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cc64ef8-1ca4-492e-950c-27be5668c276-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29\" (UID: \"7cc64ef8-1ca4-492e-950c-27be5668c276\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.119237 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-image-registry-private-configuration\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119237 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66e702f9-c995-4223-99c2-900fa238ff8a-config-volume\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.119386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-bound-sa-token\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119276 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlr74\" (UniqueName: \"kubernetes.io/projected/1a8c025e-5968-474b-8193-c3e6eb862552-kube-api-access-wlr74\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.119386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119296 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-installation-pull-secrets\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-trusted-ca\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr66q\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-kube-api-access-dr66q\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fdd52a8-81dd-4146-a49e-207380169305-cert\") pod \"ingress-canary-tw46w\" (UID: \"9fdd52a8-81dd-4146-a49e-207380169305\") " pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1a8c025e-5968-474b-8193-c3e6eb862552-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-hub\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdbs\" (UniqueName: \"kubernetes.io/projected/7cc64ef8-1ca4-492e-950c-27be5668c276-kube-api-access-4tdbs\") pod \"managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29\" (UID: \"7cc64ef8-1ca4-492e-950c-27be5668c276\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-certificates\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a8c025e-5968-474b-8193-c3e6eb862552-tmp\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0603df8f-699c-4a09-a475-67a3a2eb451f-ca-trust-extracted\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.119622 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.119613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8vc\" (UniqueName: \"kubernetes.io/projected/9fdd52a8-81dd-4146-a49e-207380169305-kube-api-access-rt8vc\") pod \"ingress-canary-tw46w\" (UID: \"9fdd52a8-81dd-4146-a49e-207380169305\") " pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.220707 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cc64ef8-1ca4-492e-950c-27be5668c276-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29\" (UID: \"7cc64ef8-1ca4-492e-950c-27be5668c276\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.220869 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-image-registry-private-configuration\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.220869 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66e702f9-c995-4223-99c2-900fa238ff8a-config-volume\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.220869 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-bound-sa-token\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.220869 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlr74\" (UniqueName: \"kubernetes.io/projected/1a8c025e-5968-474b-8193-c3e6eb862552-kube-api-access-wlr74\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.220869 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-installation-pull-secrets\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.220869 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-trusted-ca\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr66q\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-kube-api-access-dr66q\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fdd52a8-81dd-4146-a49e-207380169305-cert\") pod \"ingress-canary-tw46w\" (UID: \"9fdd52a8-81dd-4146-a49e-207380169305\") " pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1a8c025e-5968-474b-8193-c3e6eb862552-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-hub\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.220989 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdbs\" (UniqueName: \"kubernetes.io/projected/7cc64ef8-1ca4-492e-950c-27be5668c276-kube-api-access-4tdbs\") pod \"managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29\" (UID: \"7cc64ef8-1ca4-492e-950c-27be5668c276\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-certificates\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221044 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a8c025e-5968-474b-8193-c3e6eb862552-tmp\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0603df8f-699c-4a09-a475-67a3a2eb451f-ca-trust-extracted\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8vc\" (UniqueName: \"kubernetes.io/projected/9fdd52a8-81dd-4146-a49e-207380169305-kube-api-access-rt8vc\") pod \"ingress-canary-tw46w\" (UID: \"9fdd52a8-81dd-4146-a49e-207380169305\") " pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e702f9-c995-4223-99c2-900fa238ff8a-tmp-dir\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.221195 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt5s2\" (UniqueName: \"kubernetes.io/projected/66e702f9-c995-4223-99c2-900fa238ff8a-kube-api-access-rt5s2\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq42k\" (UniqueName: \"kubernetes.io/projected/4e869a5f-b91c-489d-b3be-851f22dbee7a-kube-api-access-wq42k\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-ca\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e869a5f-b91c-489d-b3be-851f22dbee7a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66e702f9-c995-4223-99c2-900fa238ff8a-config-volume\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.221772 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a8c025e-5968-474b-8193-c3e6eb862552-tmp\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.222111 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.221917 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:23.222111 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0603df8f-699c-4a09-a475-67a3a2eb451f-ca-trust-extracted\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.222111 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.221957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e869a5f-b91c-489d-b3be-851f22dbee7a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.222267 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.222247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66e702f9-c995-4223-99c2-900fa238ff8a-tmp-dir\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.223224 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.222983 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:23.722956251 +0000 UTC m=+35.584531005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:23.223224 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.222995 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:23.223224 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.223012 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:23.223224 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.223074 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:50:23.72305654 +0000 UTC m=+35.584631294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:23.223224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.223135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-certificates\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.224120 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.224028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-trusted-ca\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.226354 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.226306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1a8c025e-5968-474b-8193-c3e6eb862552-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.226472 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.226415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-image-registry-private-configuration\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.227170 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.227113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-ca\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.227170 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.227142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-installation-pull-secrets\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.227170 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.227164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-hub\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.227898 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.227770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.228071 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.228029 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7cc64ef8-1ca4-492e-950c-27be5668c276-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29\" (UID: \"7cc64ef8-1ca4-492e-950c-27be5668c276\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.228349 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.228316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4e869a5f-b91c-489d-b3be-851f22dbee7a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.228700 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.228674 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fdd52a8-81dd-4146-a49e-207380169305-cert\") pod \"ingress-canary-tw46w\" (UID: \"9fdd52a8-81dd-4146-a49e-207380169305\") " pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.233302 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.233279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8vc\" (UniqueName: \"kubernetes.io/projected/9fdd52a8-81dd-4146-a49e-207380169305-kube-api-access-rt8vc\") pod \"ingress-canary-tw46w\" (UID: \"9fdd52a8-81dd-4146-a49e-207380169305\") " pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.233662 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.233639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq42k\" (UniqueName: \"kubernetes.io/projected/4e869a5f-b91c-489d-b3be-851f22dbee7a-kube-api-access-wq42k\") pod \"cluster-proxy-proxy-agent-76dd69b9b5-6s4jj\" (UID: \"4e869a5f-b91c-489d-b3be-851f22dbee7a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.234072 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.234048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr66q\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-kube-api-access-dr66q\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.234386 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.234365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlr74\" (UniqueName: \"kubernetes.io/projected/1a8c025e-5968-474b-8193-c3e6eb862552-kube-api-access-wlr74\") pod \"klusterlet-addon-workmgr-7ddd687b7f-7w99n\" (UID: \"1a8c025e-5968-474b-8193-c3e6eb862552\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.235164 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.235106 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt5s2\" (UniqueName: \"kubernetes.io/projected/66e702f9-c995-4223-99c2-900fa238ff8a-kube-api-access-rt5s2\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.235800 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.235764 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-bound-sa-token\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.237598 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.237575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdbs\" (UniqueName: \"kubernetes.io/projected/7cc64ef8-1ca4-492e-950c-27be5668c276-kube-api-access-4tdbs\") pod \"managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29\" (UID: \"7cc64ef8-1ca4-492e-950c-27be5668c276\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.325098 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.325014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:23.340833 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.340812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:50:23.359318 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.359295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" Jul 16 21:50:23.371828 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.371805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tw46w" Jul 16 21:50:23.679345 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.679261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:23.679507 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.679261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:23.679507 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.679410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:23.686707 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.686684 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:50:23.686707 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.686699 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:50:23.688004 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.687977 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:50:23.688118 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.688011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:50:23.688118 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.688094 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqtjl\"" Jul 16 21:50:23.688427 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.688404 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xshkd\"" Jul 16 21:50:23.725202 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.725164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:23.725312 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:23.725216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:23.725358 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.725316 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:23.725358 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.725337 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:23.725358 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.725352 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:23.725458 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.725382 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:24.725365092 +0000 UTC m=+36.586939847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:23.725458 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:23.725400 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:50:24.725389978 +0000 UTC m=+36.586964722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:24.397241 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.397097 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tw46w"] Jul 16 21:50:24.401604 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.401579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29"] Jul 16 21:50:24.403564 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:50:24.403539 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdd52a8_81dd_4146_a49e_207380169305.slice/crio-4e8067b7badec77bbea8572e8fcfe83ae83e8f42206475f2d36271d29e2cf8c8 WatchSource:0}: Error finding container 4e8067b7badec77bbea8572e8fcfe83ae83e8f42206475f2d36271d29e2cf8c8: Status 404 returned error can't find the container with id 4e8067b7badec77bbea8572e8fcfe83ae83e8f42206475f2d36271d29e2cf8c8 Jul 16 21:50:24.405589 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:50:24.405571 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cc64ef8_1ca4_492e_950c_27be5668c276.slice/crio-5b5456af81036ea374477d20d371e58538febc69786128b4300931d258bbd4a3 WatchSource:0}: Error finding container 5b5456af81036ea374477d20d371e58538febc69786128b4300931d258bbd4a3: Status 404 returned error can't find the container with id 5b5456af81036ea374477d20d371e58538febc69786128b4300931d258bbd4a3 Jul 16 21:50:24.420435 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.420412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n"] Jul 16 21:50:24.424005 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.423984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj"] Jul 16 21:50:24.425003 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:50:24.424966 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8c025e_5968_474b_8193_c3e6eb862552.slice/crio-6119a317f4690cfc17a25439fee1f4e27931020b4520d853cc1b2b4cc06b2769 WatchSource:0}: Error finding container 6119a317f4690cfc17a25439fee1f4e27931020b4520d853cc1b2b4cc06b2769: Status 404 returned error can't find the container with id 6119a317f4690cfc17a25439fee1f4e27931020b4520d853cc1b2b4cc06b2769 Jul 16 21:50:24.427766 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:50:24.427746 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e869a5f_b91c_489d_b3be_851f22dbee7a.slice/crio-3473b4ae497a0832f16f50e667dc42013614244c5ed9153da1973ea5188e2177 WatchSource:0}: Error finding container 3473b4ae497a0832f16f50e667dc42013614244c5ed9153da1973ea5188e2177: Status 404 returned error can't find the container with id 3473b4ae497a0832f16f50e667dc42013614244c5ed9153da1973ea5188e2177 Jul 16 21:50:24.734012 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.733980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:24.734150 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.734024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:24.734150 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:24.734117 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:24.734150 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:24.734130 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:24.734150 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:24.734129 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:24.734293 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:24.734187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:50:26.734170175 +0000 UTC m=+38.595744934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:24.734293 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:24.734212 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:26.734205154 +0000 UTC m=+38.595779893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:24.866214 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.866129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tw46w" event={"ID":"9fdd52a8-81dd-4146-a49e-207380169305","Type":"ContainerStarted","Data":"4e8067b7badec77bbea8572e8fcfe83ae83e8f42206475f2d36271d29e2cf8c8"} Jul 16 21:50:24.867334 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.867307 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" event={"ID":"4e869a5f-b91c-489d-b3be-851f22dbee7a","Type":"ContainerStarted","Data":"3473b4ae497a0832f16f50e667dc42013614244c5ed9153da1973ea5188e2177"} Jul 16 21:50:24.870022 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.870001 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a46827d-b431-4e78-90d1-f0cdf47dff3c" containerID="4e83b443a82fbc77237c4aee7efce5298bea0a151efaab1e63fd283fd2bd1d78" exitCode=0 Jul 16 21:50:24.870090 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.870067 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerDied","Data":"4e83b443a82fbc77237c4aee7efce5298bea0a151efaab1e63fd283fd2bd1d78"} Jul 16 21:50:24.871137 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.871117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" event={"ID":"7cc64ef8-1ca4-492e-950c-27be5668c276","Type":"ContainerStarted","Data":"5b5456af81036ea374477d20d371e58538febc69786128b4300931d258bbd4a3"} Jul 16 21:50:24.872176 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:24.872156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" event={"ID":"1a8c025e-5968-474b-8193-c3e6eb862552","Type":"ContainerStarted","Data":"6119a317f4690cfc17a25439fee1f4e27931020b4520d853cc1b2b4cc06b2769"} Jul 16 21:50:25.889441 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:25.889404 2576 generic.go:356] "Generic (PLEG): container finished" podID="3a46827d-b431-4e78-90d1-f0cdf47dff3c" containerID="55911e101a626b77d6a1c229cb35ba85ecda8a2aeef9f9835d3a3dae12a6887e" exitCode=0 Jul 16 21:50:25.890084 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:25.889462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerDied","Data":"55911e101a626b77d6a1c229cb35ba85ecda8a2aeef9f9835d3a3dae12a6887e"} Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:26.751562 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:26.751736 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:26.751951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:26.752001 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:30.751977791 +0000 UTC m=+42.613552534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:26.752089 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:26.752106 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:26.752229 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:26.752239 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:50:30.752220062 +0000 UTC m=+42.613794823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:26.899787 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:26.899750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" event={"ID":"3a46827d-b431-4e78-90d1-f0cdf47dff3c","Type":"ContainerStarted","Data":"1bb3be6be51c3a931c3c17a32ecc3db6776b13aebde12ae9f9dc43efa5b1b621"} Jul 16 21:50:26.930140 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:26.929829 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s9v8c" podStartSLOduration=9.434926295 podStartE2EDuration="38.929806893s" totalImagesPullingTime="29.494880598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:26.928115844 +0000 UTC m=+38.789690608" watchObservedRunningTime="2026-07-16 21:50:26.929806893 +0000 UTC m=+38.791381654" Jul 16 21:50:30.787864 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:30.787824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:30.788308 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:30.787971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:30.788308 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:30.788062 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:30.788308 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:30.788087 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:30.788308 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:30.788149 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.788129156 +0000 UTC m=+50.649703914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:30.788308 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:30.788068 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:30.788308 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:30.788222 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.788206695 +0000 UTC m=+50.649781449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:32.917526 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.917489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" event={"ID":"7cc64ef8-1ca4-492e-950c-27be5668c276","Type":"ContainerStarted","Data":"7bda8ed0482779df9be25c0d30e152507aa8c9e3813966669c149b6bff62b7fe"} Jul 16 21:50:32.919001 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.918965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" event={"ID":"1a8c025e-5968-474b-8193-c3e6eb862552","Type":"ContainerStarted","Data":"39c3505b784d06f13ae6a2463fefbef1f78a1619cdb76d5d594726a40cf77272"} Jul 16 21:50:32.919138 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.919108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:32.920499 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.920464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tw46w" event={"ID":"9fdd52a8-81dd-4146-a49e-207380169305","Type":"ContainerStarted","Data":"e1d1be6ed5c6c95fc2eb6de2ddc18e3f9464a364c8900d0a80ce1fa4627bad46"} Jul 16 21:50:32.921178 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.921159 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" Jul 16 21:50:32.922688 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.922666 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" event={"ID":"4e869a5f-b91c-489d-b3be-851f22dbee7a","Type":"ContainerStarted","Data":"0a334e4b7166128c1079b2a20fe140bb495b7ab40a2dda68f52b32f95fee24f6"} Jul 16 21:50:32.922778 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.922701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" event={"ID":"4e869a5f-b91c-489d-b3be-851f22dbee7a","Type":"ContainerStarted","Data":"359e806ffa0e5736b5341332fa2baf114e4cdffa216dd0e59f74398ebc6d54da"} Jul 16 21:50:32.922778 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.922717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" event={"ID":"4e869a5f-b91c-489d-b3be-851f22dbee7a","Type":"ContainerStarted","Data":"918ec1eb6b7b567cfb39db5a4b02776b18cd4e342a6a8fe6a469725062427cb4"} Jul 16 21:50:32.934811 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.934767 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f98f6c8c6-4vh29" podStartSLOduration=31.465064489 podStartE2EDuration="38.934752005s" totalImagesPullingTime="7.469687516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:32.934458089 +0000 UTC m=+44.796032853" watchObservedRunningTime="2026-07-16 21:50:32.934752005 +0000 UTC m=+44.796326780" Jul 16 21:50:32.953596 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.953561 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" podStartSLOduration=31.166692564 podStartE2EDuration="38.953549547s" totalImagesPullingTime="7.786856983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:32.952952613 +0000 UTC m=+44.814527375" watchObservedRunningTime="2026-07-16 21:50:32.953549547 +0000 UTC m=+44.815124350" Jul 16 21:50:32.969349 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.969316 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd687b7f-7w99n" podStartSLOduration=31.49983882 podStartE2EDuration="38.969303859s" totalImagesPullingTime="7.469465039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:32.96889776 +0000 UTC m=+44.830472521" watchObservedRunningTime="2026-07-16 21:50:32.969303859 +0000 UTC m=+44.830878621" Jul 16 21:50:32.984086 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:32.984051 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tw46w" podStartSLOduration=3.5115388320000003 podStartE2EDuration="10.984038551s" totalImagesPullingTime="7.472499719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:32.983878419 +0000 UTC m=+44.845453184" watchObservedRunningTime="2026-07-16 21:50:32.984038551 +0000 UTC m=+44.845613314" Jul 16 21:50:38.848982 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:38.848946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:38.849422 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:38.849045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:38.849422 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:38.849103 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:38.849422 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:38.849118 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:38.849422 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:38.849124 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:38.849422 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:38.849178 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:50:54.84916212 +0000 UTC m=+66.710736876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:38.849422 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:38.849193 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:50:54.849187208 +0000 UTC m=+66.710761980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:47.857073 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:47.857045 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-564jb" Jul 16 21:50:54.366741 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.366694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:50:54.369963 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.369938 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 21:50:54.377428 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.377413 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 16 21:50:54.377484 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.377478 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs podName:5914b657-8a03-4f03-90d5-b94f1bb5301a nodeName:}" failed. No retries permitted until 2026-07-16 21:51:58.37746339 +0000 UTC m=+130.239038131 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs") pod "network-metrics-daemon-27sgl" (UID: "5914b657-8a03-4f03-90d5-b94f1bb5301a") : secret "metrics-daemon-secret" not found Jul 16 21:50:54.467975 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.467948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:54.468068 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.467997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:54.471618 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.471600 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 21:50:54.471682 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.471643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 21:50:54.482164 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.482148 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 21:50:54.482797 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.482777 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f051cc1e-0917-46a4-bf69-e69da8dd4b65-original-pull-secret\") pod \"global-pull-secret-syncer-qjjxj\" (UID: \"f051cc1e-0917-46a4-bf69-e69da8dd4b65\") " pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:54.491587 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.491567 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q8j7\" (UniqueName: \"kubernetes.io/projected/92104d61-3706-4d0c-beb9-ce3905eb8740-kube-api-access-7q8j7\") pod \"network-check-target-lc22p\" (UID: \"92104d61-3706-4d0c-beb9-ce3905eb8740\") " pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:54.591604 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.591580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qjjxj" Jul 16 21:50:54.606642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.606610 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqtjl\"" Jul 16 21:50:54.614369 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.614352 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:50:54.725817 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.725784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qjjxj"] Jul 16 21:50:54.728741 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:50:54.728713 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf051cc1e_0917_46a4_bf69_e69da8dd4b65.slice/crio-4c87c928aabd293a1908159415264b2adbc9bd558d54b60edac695a617eaf5bd WatchSource:0}: Error finding container 4c87c928aabd293a1908159415264b2adbc9bd558d54b60edac695a617eaf5bd: Status 404 returned error can't find the container with id 4c87c928aabd293a1908159415264b2adbc9bd558d54b60edac695a617eaf5bd Jul 16 21:50:54.743369 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.743345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lc22p"] Jul 16 21:50:54.746133 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:50:54.746111 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92104d61_3706_4d0c_beb9_ce3905eb8740.slice/crio-649fcc63adda26ac9e95ffa52db9b34675bc11b65c16d5081a6236958e8d6c3e WatchSource:0}: Error finding container 649fcc63adda26ac9e95ffa52db9b34675bc11b65c16d5081a6236958e8d6c3e: Status 404 returned error can't find the container with id 649fcc63adda26ac9e95ffa52db9b34675bc11b65c16d5081a6236958e8d6c3e Jul 16 21:50:54.870748 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.870720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:50:54.870866 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.870753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:50:54.870866 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.870862 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 21:50:54.870999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.870872 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57749b7b4c-6wtsb: secret "image-registry-tls" not found Jul 16 21:50:54.870999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.870873 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 21:50:54.870999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.870916 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls podName:0603df8f-699c-4a09-a475-67a3a2eb451f nodeName:}" failed. No retries permitted until 2026-07-16 21:51:26.870902835 +0000 UTC m=+98.732477575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls") pod "image-registry-57749b7b4c-6wtsb" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f") : secret "image-registry-tls" not found Jul 16 21:50:54.870999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:50:54.870946 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls podName:66e702f9-c995-4223-99c2-900fa238ff8a nodeName:}" failed. No retries permitted until 2026-07-16 21:51:26.870922746 +0000 UTC m=+98.732497486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls") pod "dns-default-588zz" (UID: "66e702f9-c995-4223-99c2-900fa238ff8a") : secret "dns-default-metrics-tls" not found Jul 16 21:50:54.989708 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.989673 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lc22p" event={"ID":"92104d61-3706-4d0c-beb9-ce3905eb8740","Type":"ContainerStarted","Data":"649fcc63adda26ac9e95ffa52db9b34675bc11b65c16d5081a6236958e8d6c3e"} Jul 16 21:50:54.990794 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:50:54.990772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qjjxj" event={"ID":"f051cc1e-0917-46a4-bf69-e69da8dd4b65","Type":"ContainerStarted","Data":"4c87c928aabd293a1908159415264b2adbc9bd558d54b60edac695a617eaf5bd"} Jul 16 21:51:00.009910 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:00.009870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qjjxj" event={"ID":"f051cc1e-0917-46a4-bf69-e69da8dd4b65","Type":"ContainerStarted","Data":"382f5a0c5bb3fb2321a9023d31a7dfb967f3948d45e2ae77f9fe354c1c86dde5"} Jul 16 21:51:00.011528 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:00.011502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lc22p" event={"ID":"92104d61-3706-4d0c-beb9-ce3905eb8740","Type":"ContainerStarted","Data":"908049c297a07b33aa61214308d50cf1452015de71daa85126c86498578d21d8"} Jul 16 21:51:00.011652 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:00.011572 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:51:00.033481 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:00.033431 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qjjxj" podStartSLOduration=66.240025685 podStartE2EDuration="1m11.033416225s" totalImagesPullingTime="4.79339054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:00.03150485 +0000 UTC m=+71.893079611" watchObservedRunningTime="2026-07-16 21:51:00.033416225 +0000 UTC m=+71.894990989" Jul 16 21:51:04.034170 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.034121 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lc22p" podStartSLOduration=70.263576374 podStartE2EDuration="1m15.034104267s" totalImagesPullingTime="4.770527893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:00.059368907 +0000 UTC m=+71.920943668" watchObservedRunningTime="2026-07-16 21:51:04.034104267 +0000 UTC m=+75.895679007" Jul 16 21:51:04.034657 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.034642 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qvrlv"] Jul 16 21:51:04.083009 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.082981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qvrlv"] Jul 16 21:51:04.083149 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.083079 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.087439 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.087421 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 21:51:04.089535 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.089515 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 21:51:04.089535 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.089524 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 21:51:04.089682 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.089585 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 21:51:04.091757 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.091741 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8tffc\"" Jul 16 21:51:04.144281 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.144252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-crio-socket\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.144281 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.144285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-data-volume\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.144474 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.144312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.144474 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.144408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.144474 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.144453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blm4h\" (UniqueName: \"kubernetes.io/projected/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-api-access-blm4h\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.244736 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.244699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blm4h\" (UniqueName: \"kubernetes.io/projected/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-api-access-blm4h\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.244886 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.244770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-crio-socket\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.244886 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.244796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-data-volume\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.244886 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.244817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.244886 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.244853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.245059 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.244906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-crio-socket\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.245059 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.244951 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:04.245059 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.244969 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.245059 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.245012 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.744999279 +0000 UTC m=+76.606574031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : configmap "kube-rbac-proxy" not found Jul 16 21:51:04.245059 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.245025 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.745019664 +0000 UTC m=+76.606594404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.245221 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.245190 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-data-volume\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.277448 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.277418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blm4h\" (UniqueName: \"kubernetes.io/projected/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-api-access-blm4h\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.748042 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.748007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.748223 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:04.748094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:04.748223 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.748122 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:04.748223 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.748185 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.748170063 +0000 UTC m=+77.609744803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : configmap "kube-rbac-proxy" not found Jul 16 21:51:04.748223 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.748190 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:04.748358 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:04.748231 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.748218763 +0000 UTC m=+77.609793503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:05.755535 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:05.755497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:05.755999 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:05.755549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:05.755999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:05.755633 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:05.755999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:05.755694 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.755679343 +0000 UTC m=+79.617254083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:05.755999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:05.755635 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:05.755999 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:05.755785 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.755768096 +0000 UTC m=+79.617342843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : configmap "kube-rbac-proxy" not found Jul 16 21:51:06.831865 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:06.831835 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dtt29_bc099ed7-cc93-4665-9d00-35bc13b47a3c/dns-node-resolver/0.log" Jul 16 21:51:07.774231 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:07.774190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:07.774440 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:07.774274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:07.774440 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:07.774325 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:07.774440 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:07.774362 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:07.774440 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:07.774395 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:11.774378398 +0000 UTC m=+83.635953152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : configmap "kube-rbac-proxy" not found Jul 16 21:51:07.774440 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:07.774422 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:11.774404416 +0000 UTC m=+83.635979168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:08.029426 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:08.029348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lt998_e4d190f3-cbee-4efd-b727-fd11a101fc52/node-ca/0.log" Jul 16 21:51:08.629243 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:08.629208 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tw46w_9fdd52a8-81dd-4146-a49e-207380169305/serve-healthcheck-canary/0.log" Jul 16 21:51:11.807102 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:11.806975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:11.807102 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:11.807022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:11.807507 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:11.807125 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 21:51:11.807507 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:11.807175 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:19.807160697 +0000 UTC m=+91.668735451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : configmap "kube-rbac-proxy" not found Jul 16 21:51:11.807507 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:11.807125 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 21:51:11.807507 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:51:11.807282 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls podName:53ca1d22-b598-4c7c-8f8f-9f675f9dbff8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:19.80726051 +0000 UTC m=+91.668835250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qvrlv" (UID: "53ca1d22-b598-4c7c-8f8f-9f675f9dbff8") : secret "insights-runtime-extractor-tls" not found Jul 16 21:51:19.869147 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:19.869108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:19.869558 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:19.869164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:19.869730 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:19.869709 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:19.871469 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:19.871450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53ca1d22-b598-4c7c-8f8f-9f675f9dbff8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qvrlv\" (UID: \"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8\") " pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:19.991536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:19.991504 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qvrlv" Jul 16 21:51:20.127551 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:20.127515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qvrlv"] Jul 16 21:51:20.129707 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:51:20.129667 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ca1d22_b598_4c7c_8f8f_9f675f9dbff8.slice/crio-9ab006285874cef72ab8d52103c7bfae62ea44139aa2f9068d52e60ea5541185 WatchSource:0}: Error finding container 9ab006285874cef72ab8d52103c7bfae62ea44139aa2f9068d52e60ea5541185: Status 404 returned error can't find the container with id 9ab006285874cef72ab8d52103c7bfae62ea44139aa2f9068d52e60ea5541185 Jul 16 21:51:21.080774 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:21.080738 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvrlv" event={"ID":"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8","Type":"ContainerStarted","Data":"9ab006285874cef72ab8d52103c7bfae62ea44139aa2f9068d52e60ea5541185"} Jul 16 21:51:26.100760 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.100717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvrlv" event={"ID":"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8","Type":"ContainerStarted","Data":"ad8e8f216d67c24b23317cb02305421d5b6b6915f52877b2b023386cee625a5a"} Jul 16 21:51:26.927822 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.927777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:51:26.928038 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.927871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:51:26.930540 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.930511 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66e702f9-c995-4223-99c2-900fa238ff8a-metrics-tls\") pod \"dns-default-588zz\" (UID: \"66e702f9-c995-4223-99c2-900fa238ff8a\") " pod="openshift-dns/dns-default-588zz" Jul 16 21:51:26.930540 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.930533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"image-registry-57749b7b4c-6wtsb\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:51:26.968549 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.968515 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n6rt\"" Jul 16 21:51:26.976027 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:26.975994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-588zz" Jul 16 21:51:27.220180 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:27.220152 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nmm9q\"" Jul 16 21:51:27.227455 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:27.227427 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:51:28.766833 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:28.766796 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57749b7b4c-6wtsb"] Jul 16 21:51:28.771293 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:51:28.771265 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0603df8f_699c_4a09_a475_67a3a2eb451f.slice/crio-7a2e5fcc919774c69385dc9c9b7c42855da271cddb561de7158d28f547b63f53 WatchSource:0}: Error finding container 7a2e5fcc919774c69385dc9c9b7c42855da271cddb561de7158d28f547b63f53: Status 404 returned error can't find the container with id 7a2e5fcc919774c69385dc9c9b7c42855da271cddb561de7158d28f547b63f53 Jul 16 21:51:28.786389 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:28.786361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-588zz"] Jul 16 21:51:28.790813 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:51:28.790781 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e702f9_c995_4223_99c2_900fa238ff8a.slice/crio-21fac295c15c9c641f91af0cfa790270cea55b6f5fe4a101518c7660f51436b0 WatchSource:0}: Error finding container 21fac295c15c9c641f91af0cfa790270cea55b6f5fe4a101518c7660f51436b0: Status 404 returned error can't find the container with id 21fac295c15c9c641f91af0cfa790270cea55b6f5fe4a101518c7660f51436b0 Jul 16 21:51:29.111734 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:29.111693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-588zz" event={"ID":"66e702f9-c995-4223-99c2-900fa238ff8a","Type":"ContainerStarted","Data":"21fac295c15c9c641f91af0cfa790270cea55b6f5fe4a101518c7660f51436b0"} Jul 16 21:51:29.113497 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:29.113467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvrlv" event={"ID":"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8","Type":"ContainerStarted","Data":"0aa03d474555f4fcbbeecab15854a50d8f58f8ce6b7ed59e43570ac51ddd1586"} Jul 16 21:51:29.114873 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:29.114853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" event={"ID":"0603df8f-699c-4a09-a475-67a3a2eb451f","Type":"ContainerStarted","Data":"94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955"} Jul 16 21:51:29.114986 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:29.114892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" event={"ID":"0603df8f-699c-4a09-a475-67a3a2eb451f","Type":"ContainerStarted","Data":"7a2e5fcc919774c69385dc9c9b7c42855da271cddb561de7158d28f547b63f53"} Jul 16 21:51:29.140075 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:29.140025 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" podStartSLOduration=100.140007535 podStartE2EDuration="1m40.140007535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:29.138525163 +0000 UTC m=+101.000099937" watchObservedRunningTime="2026-07-16 21:51:29.140007535 +0000 UTC m=+101.001582296" Jul 16 21:51:30.119379 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:30.119314 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:51:31.017533 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:31.017496 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lc22p" Jul 16 21:51:33.134740 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:33.134696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qvrlv" event={"ID":"53ca1d22-b598-4c7c-8f8f-9f675f9dbff8","Type":"ContainerStarted","Data":"234b5eaaada88ec08b5698a594e09bfcf3e613ed085d9a48a57d9c9627e1c7ab"} Jul 16 21:51:33.136389 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:33.136359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-588zz" event={"ID":"66e702f9-c995-4223-99c2-900fa238ff8a","Type":"ContainerStarted","Data":"800e1a49fd055047a09e5e6574606697cfa6d4cda266315455a0b624b774fa64"} Jul 16 21:51:33.136505 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:33.136413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-588zz" event={"ID":"66e702f9-c995-4223-99c2-900fa238ff8a","Type":"ContainerStarted","Data":"4603c96f540874fd777fe1e19ae22c009ff0457177e9acab132e65d07c33c9fb"} Jul 16 21:51:33.136569 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:33.136509 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-588zz" Jul 16 21:51:33.156002 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:33.155955 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qvrlv" podStartSLOduration=17.895712767 podStartE2EDuration="30.15592026s" totalImagesPullingTime="12.260207493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:33.155516518 +0000 UTC m=+105.017091279" watchObservedRunningTime="2026-07-16 21:51:33.15592026 +0000 UTC m=+105.017495023" Jul 16 21:51:33.178452 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:33.178400 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-588zz" podStartSLOduration=67.468683784 podStartE2EDuration="1m11.17838287s" totalImagesPullingTime="3.709699086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:33.177038199 +0000 UTC m=+105.038612961" watchObservedRunningTime="2026-07-16 21:51:33.17838287 +0000 UTC m=+105.039957631" Jul 16 21:51:40.561520 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.561487 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-79z9k"] Jul 16 21:51:40.564642 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.564622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.571766 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.571747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 21:51:40.571904 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.571780 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 21:51:40.571904 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.571863 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 21:51:40.571904 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.571868 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-drdzb\"" Jul 16 21:51:40.572290 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.572275 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 21:51:40.572290 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.572283 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 21:51:40.572414 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.572303 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 21:51:40.638191 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-root\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638321 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-textfile\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638321 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-wtmp\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638321 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c712b2f-3057-4593-a4f2-4fdc48eccd36-metrics-client-ca\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638321 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638447 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-accelerators-collector-config\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638447 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbswp\" (UniqueName: \"kubernetes.io/projected/4c712b2f-3057-4593-a4f2-4fdc48eccd36-kube-api-access-rbswp\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638447 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-tls\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.638447 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.638383 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-sys\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738634 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-textfile\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738634 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-wtmp\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c712b2f-3057-4593-a4f2-4fdc48eccd36-metrics-client-ca\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-accelerators-collector-config\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbswp\" (UniqueName: \"kubernetes.io/projected/4c712b2f-3057-4593-a4f2-4fdc48eccd36-kube-api-access-rbswp\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-tls\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-sys\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738810 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-wtmp\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.738842 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738841 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-sys\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.739282 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-root\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.739282 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.738956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c712b2f-3057-4593-a4f2-4fdc48eccd36-root\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.739282 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.739019 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-textfile\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.739440 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.739384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c712b2f-3057-4593-a4f2-4fdc48eccd36-metrics-client-ca\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.739490 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.739436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-accelerators-collector-config\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.740948 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.740907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.741049 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.741037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c712b2f-3057-4593-a4f2-4fdc48eccd36-node-exporter-tls\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.758230 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.758202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbswp\" (UniqueName: \"kubernetes.io/projected/4c712b2f-3057-4593-a4f2-4fdc48eccd36-kube-api-access-rbswp\") pod \"node-exporter-79z9k\" (UID: \"4c712b2f-3057-4593-a4f2-4fdc48eccd36\") " pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.873280 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:40.873218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-79z9k" Jul 16 21:51:40.883128 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:51:40.883106 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c712b2f_3057_4593_a4f2_4fdc48eccd36.slice/crio-1a3f2ad34f7214a4d8db680a2bae3e4077c24ef3fca27df85cc6c743ea1ac1a0 WatchSource:0}: Error finding container 1a3f2ad34f7214a4d8db680a2bae3e4077c24ef3fca27df85cc6c743ea1ac1a0: Status 404 returned error can't find the container with id 1a3f2ad34f7214a4d8db680a2bae3e4077c24ef3fca27df85cc6c743ea1ac1a0 Jul 16 21:51:41.164013 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:41.163914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-79z9k" event={"ID":"4c712b2f-3057-4593-a4f2-4fdc48eccd36","Type":"ContainerStarted","Data":"1a3f2ad34f7214a4d8db680a2bae3e4077c24ef3fca27df85cc6c743ea1ac1a0"} Jul 16 21:51:42.168339 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:42.168254 2576 generic.go:356] "Generic (PLEG): container finished" podID="4c712b2f-3057-4593-a4f2-4fdc48eccd36" containerID="17621bef3035928658a6cc394fcea959ba3a0368f99dbd2fb3a4965cf68e8c78" exitCode=0 Jul 16 21:51:42.168704 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:42.168350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-79z9k" event={"ID":"4c712b2f-3057-4593-a4f2-4fdc48eccd36","Type":"ContainerDied","Data":"17621bef3035928658a6cc394fcea959ba3a0368f99dbd2fb3a4965cf68e8c78"} Jul 16 21:51:43.142481 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:43.142452 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-588zz" Jul 16 21:51:43.172989 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:43.172958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-79z9k" event={"ID":"4c712b2f-3057-4593-a4f2-4fdc48eccd36","Type":"ContainerStarted","Data":"d67c614a872b91253c2257ca24fa1ad0a4c2b581a55596e432e4f75debeebade"} Jul 16 21:51:43.173328 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:43.173003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-79z9k" event={"ID":"4c712b2f-3057-4593-a4f2-4fdc48eccd36","Type":"ContainerStarted","Data":"344896d3dad98a91d97b667b98bdc20cbb327ec98bf349bfd99fd20ffa92969f"} Jul 16 21:51:43.202706 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:43.202663 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-79z9k" podStartSLOduration=2.262496813 podStartE2EDuration="3.202650022s" totalImagesPullingTime="877.755142ms" totalInitContainerRuntime="62.398067ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:43.201734982 +0000 UTC m=+115.063309745" watchObservedRunningTime="2026-07-16 21:51:43.202650022 +0000 UTC m=+115.064224783" Jul 16 21:51:47.231597 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:47.231564 2576 patch_prober.go:28] interesting pod/image-registry-57749b7b4c-6wtsb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 16 21:51:47.232002 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:47.231623 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" podUID="0603df8f-699c-4a09-a475-67a3a2eb451f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 21:51:51.127358 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:51.127327 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:51:53.342502 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:53.342463 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" podUID="4e869a5f-b91c-489d-b3be-851f22dbee7a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 16 21:51:53.782381 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:53.782344 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57749b7b4c-6wtsb"] Jul 16 21:51:58.473224 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:58.473190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:51:58.475423 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:58.475403 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5914b657-8a03-4f03-90d5-b94f1bb5301a-metrics-certs\") pod \"network-metrics-daemon-27sgl\" (UID: \"5914b657-8a03-4f03-90d5-b94f1bb5301a\") " pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:51:58.501314 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:58.501290 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xshkd\"" Jul 16 21:51:58.509708 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:58.509690 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-27sgl" Jul 16 21:51:58.652329 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:58.652301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-27sgl"] Jul 16 21:51:58.655980 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:51:58.655951 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5914b657_8a03_4f03_90d5_b94f1bb5301a.slice/crio-753e0c42467c28f20e2b3ae3cd872d99cc2ace37e31e602c961211a2055cc1fc WatchSource:0}: Error finding container 753e0c42467c28f20e2b3ae3cd872d99cc2ace37e31e602c961211a2055cc1fc: Status 404 returned error can't find the container with id 753e0c42467c28f20e2b3ae3cd872d99cc2ace37e31e602c961211a2055cc1fc Jul 16 21:51:59.222983 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:51:59.222943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-27sgl" event={"ID":"5914b657-8a03-4f03-90d5-b94f1bb5301a","Type":"ContainerStarted","Data":"753e0c42467c28f20e2b3ae3cd872d99cc2ace37e31e602c961211a2055cc1fc"} Jul 16 21:52:00.228276 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:00.228238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-27sgl" event={"ID":"5914b657-8a03-4f03-90d5-b94f1bb5301a","Type":"ContainerStarted","Data":"8656220e29cd9df0253cfb77c2fa549e59029a9bea14a34d6aae33deb7856fc9"} Jul 16 21:52:00.228663 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:00.228297 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-27sgl" event={"ID":"5914b657-8a03-4f03-90d5-b94f1bb5301a","Type":"ContainerStarted","Data":"2235607d864874c7ef7806c8ebddabe706fbed2fe6b3a400a7a234b4c8d61453"} Jul 16 21:52:00.246853 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:00.246812 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-27sgl" podStartSLOduration=131.261646014 podStartE2EDuration="2m12.246796593s" totalImagesPullingTime="985.150579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:00.245408377 +0000 UTC m=+132.106983138" watchObservedRunningTime="2026-07-16 21:52:00.246796593 +0000 UTC m=+132.108371356" Jul 16 21:52:03.342867 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:03.342818 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" podUID="4e869a5f-b91c-489d-b3be-851f22dbee7a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 16 21:52:13.342369 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:13.342308 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" podUID="4e869a5f-b91c-489d-b3be-851f22dbee7a" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 16 21:52:13.342751 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:13.342415 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" Jul 16 21:52:13.343017 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:13.342985 2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0a334e4b7166128c1079b2a20fe140bb495b7ab40a2dda68f52b32f95fee24f6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 16 21:52:13.343061 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:13.343042 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" podUID="4e869a5f-b91c-489d-b3be-851f22dbee7a" containerName="service-proxy" containerID="cri-o://0a334e4b7166128c1079b2a20fe140bb495b7ab40a2dda68f52b32f95fee24f6" gracePeriod=30 Jul 16 21:52:14.278983 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:14.278940 2576 generic.go:356] "Generic (PLEG): container finished" podID="4e869a5f-b91c-489d-b3be-851f22dbee7a" containerID="0a334e4b7166128c1079b2a20fe140bb495b7ab40a2dda68f52b32f95fee24f6" exitCode=2 Jul 16 21:52:14.279148 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:14.279003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" event={"ID":"4e869a5f-b91c-489d-b3be-851f22dbee7a","Type":"ContainerDied","Data":"0a334e4b7166128c1079b2a20fe140bb495b7ab40a2dda68f52b32f95fee24f6"} Jul 16 21:52:14.279148 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:14.279052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76dd69b9b5-6s4jj" event={"ID":"4e869a5f-b91c-489d-b3be-851f22dbee7a","Type":"ContainerStarted","Data":"8efa2cfcd008660fa0fbb06d12de66510ae823bf0d2e252246cc567b09df6453"} Jul 16 21:52:18.801564 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:18.801522 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" podUID="0603df8f-699c-4a09-a475-67a3a2eb451f" containerName="registry" containerID="cri-o://94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955" gracePeriod=30 Jul 16 21:52:19.051696 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.051641 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:52:19.124476 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124448 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-bound-sa-token\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124597 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124482 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-certificates\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124597 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124503 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-installation-pull-secrets\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124597 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124540 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr66q\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-kube-api-access-dr66q\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124597 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124584 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124792 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124611 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-image-registry-private-configuration\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124792 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124647 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-trusted-ca\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124792 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124669 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0603df8f-699c-4a09-a475-67a3a2eb451f-ca-trust-extracted\") pod \"0603df8f-699c-4a09-a475-67a3a2eb451f\" (UID: \"0603df8f-699c-4a09-a475-67a3a2eb451f\") " Jul 16 21:52:19.124983 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.124872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:19.126789 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.125742 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 21:52:19.128958 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.127317 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:52:19.131033 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.130988 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:52:19.131338 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.131154 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-kube-api-access-dr66q" (OuterVolumeSpecName: "kube-api-access-dr66q") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "kube-api-access-dr66q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:52:19.131338 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.131201 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 21:52:19.131338 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.131241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 21:52:19.137745 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.137719 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0603df8f-699c-4a09-a475-67a3a2eb451f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0603df8f-699c-4a09-a475-67a3a2eb451f" (UID: "0603df8f-699c-4a09-a475-67a3a2eb451f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 21:52:19.225716 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225694 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-tls\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225716 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225715 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-image-registry-private-configuration\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225726 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-trusted-ca\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225735 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0603df8f-699c-4a09-a475-67a3a2eb451f-ca-trust-extracted\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225743 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-bound-sa-token\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225751 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0603df8f-699c-4a09-a475-67a3a2eb451f-registry-certificates\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225760 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0603df8f-699c-4a09-a475-67a3a2eb451f-installation-pull-secrets\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.225845 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.225768 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr66q\" (UniqueName: \"kubernetes.io/projected/0603df8f-699c-4a09-a475-67a3a2eb451f-kube-api-access-dr66q\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 21:52:19.296536 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.296507 2576 generic.go:356] "Generic (PLEG): container finished" podID="0603df8f-699c-4a09-a475-67a3a2eb451f" containerID="94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955" exitCode=0 Jul 16 21:52:19.296633 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.296568 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" Jul 16 21:52:19.296671 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.296563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" event={"ID":"0603df8f-699c-4a09-a475-67a3a2eb451f","Type":"ContainerDied","Data":"94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955"} Jul 16 21:52:19.296729 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.296714 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57749b7b4c-6wtsb" event={"ID":"0603df8f-699c-4a09-a475-67a3a2eb451f","Type":"ContainerDied","Data":"7a2e5fcc919774c69385dc9c9b7c42855da271cddb561de7158d28f547b63f53"} Jul 16 21:52:19.296764 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.296747 2576 scope.go:117] "RemoveContainer" containerID="94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955" Jul 16 21:52:19.305480 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.305463 2576 scope.go:117] "RemoveContainer" containerID="94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955" Jul 16 21:52:19.305724 ip-10-0-128-250 kubenswrapper[2576]: E0716 21:52:19.305704 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955\": container with ID starting with 94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955 not found: ID does not exist" containerID="94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955" Jul 16 21:52:19.305768 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.305732 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955"} err="failed to get container status \"94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955\": rpc error: code = NotFound desc = could not find container \"94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955\": container with ID starting with 94edfec81f7b9480a59ef26e95bf7a74a19fc9073aa073a9e1faea5f31623955 not found: ID does not exist" Jul 16 21:52:19.318354 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.318331 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57749b7b4c-6wtsb"] Jul 16 21:52:19.320067 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:19.320048 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57749b7b4c-6wtsb"] Jul 16 21:52:20.683761 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:52:20.683731 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0603df8f-699c-4a09-a475-67a3a2eb451f" path="/var/lib/kubelet/pods/0603df8f-699c-4a09-a475-67a3a2eb451f/volumes" Jul 16 21:54:48.551710 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:54:48.551683 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 21:57:13.591135 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.591043 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-mhhzq"] Jul 16 21:57:13.591701 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.591402 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0603df8f-699c-4a09-a475-67a3a2eb451f" containerName="registry" Jul 16 21:57:13.591701 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.591419 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0603df8f-699c-4a09-a475-67a3a2eb451f" containerName="registry" Jul 16 21:57:13.591701 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.591487 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0603df8f-699c-4a09-a475-67a3a2eb451f" containerName="registry" Jul 16 21:57:13.594282 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.594261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.597616 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.597583 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 16 21:57:13.597739 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.597624 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 16 21:57:13.597739 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.597650 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 16 21:57:13.598855 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.598839 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-hsxtn\"" Jul 16 21:57:13.612518 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.612491 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-mhhzq"] Jul 16 21:57:13.745722 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.745683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tn2l\" (UniqueName: \"kubernetes.io/projected/f4810e2b-10d8-4260-b62f-765bd9cdcddb-kube-api-access-2tn2l\") pod \"llmisvc-controller-manager-654c557856-mhhzq\" (UID: \"f4810e2b-10d8-4260-b62f-765bd9cdcddb\") " pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.745901 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.745768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4810e2b-10d8-4260-b62f-765bd9cdcddb-cert\") pod \"llmisvc-controller-manager-654c557856-mhhzq\" (UID: \"f4810e2b-10d8-4260-b62f-765bd9cdcddb\") " pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.846165 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.846089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tn2l\" (UniqueName: \"kubernetes.io/projected/f4810e2b-10d8-4260-b62f-765bd9cdcddb-kube-api-access-2tn2l\") pod \"llmisvc-controller-manager-654c557856-mhhzq\" (UID: \"f4810e2b-10d8-4260-b62f-765bd9cdcddb\") " pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.846330 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.846172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4810e2b-10d8-4260-b62f-765bd9cdcddb-cert\") pod \"llmisvc-controller-manager-654c557856-mhhzq\" (UID: \"f4810e2b-10d8-4260-b62f-765bd9cdcddb\") " pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.848543 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.848519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4810e2b-10d8-4260-b62f-765bd9cdcddb-cert\") pod \"llmisvc-controller-manager-654c557856-mhhzq\" (UID: \"f4810e2b-10d8-4260-b62f-765bd9cdcddb\") " pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.856054 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.856031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tn2l\" (UniqueName: \"kubernetes.io/projected/f4810e2b-10d8-4260-b62f-765bd9cdcddb-kube-api-access-2tn2l\") pod \"llmisvc-controller-manager-654c557856-mhhzq\" (UID: \"f4810e2b-10d8-4260-b62f-765bd9cdcddb\") " pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:13.903542 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:13.903517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:14.029442 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:14.029406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-mhhzq"] Jul 16 21:57:14.032164 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:57:14.032128 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4810e2b_10d8_4260_b62f_765bd9cdcddb.slice/crio-3dc3640d482d054adc217320e529fcee4ad40257e753bcac741fcfa91e056ac1 WatchSource:0}: Error finding container 3dc3640d482d054adc217320e529fcee4ad40257e753bcac741fcfa91e056ac1: Status 404 returned error can't find the container with id 3dc3640d482d054adc217320e529fcee4ad40257e753bcac741fcfa91e056ac1 Jul 16 21:57:14.033596 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:14.033575 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 21:57:14.210841 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:14.210765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" event={"ID":"f4810e2b-10d8-4260-b62f-765bd9cdcddb","Type":"ContainerStarted","Data":"3dc3640d482d054adc217320e529fcee4ad40257e753bcac741fcfa91e056ac1"} Jul 16 21:57:17.226522 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:17.226478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" event={"ID":"f4810e2b-10d8-4260-b62f-765bd9cdcddb","Type":"ContainerStarted","Data":"3e3832a9974fde735d8c08a1e90dd3c853c96dafd15dfe570d7d7a40ae2dd3d7"} Jul 16 21:57:17.226949 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:17.226634 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:57:17.252723 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:17.252674 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" podStartSLOduration=1.841584984 podStartE2EDuration="4.252657344s" totalImagesPullingTime="2.41107236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:17.251288507 +0000 UTC m=+449.112863270" watchObservedRunningTime="2026-07-16 21:57:17.252657344 +0000 UTC m=+449.114232106" Jul 16 21:57:48.231616 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:57:48.231582 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-mhhzq" Jul 16 21:58:30.414551 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.414514 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-99mfz"] Jul 16 21:58:30.417466 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.417448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.421084 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.421063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-qt5ls\"" Jul 16 21:58:30.422426 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.422404 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jul 16 21:58:30.437430 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.437406 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-h7qgp"] Jul 16 21:58:30.440646 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.440623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-99mfz"] Jul 16 21:58:30.440775 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.440753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.443667 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.443648 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-82f2d\"" Jul 16 21:58:30.443667 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.443664 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jul 16 21:58:30.461881 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.461856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqnf8\" (UniqueName: \"kubernetes.io/projected/f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9-kube-api-access-pqnf8\") pod \"model-serving-api-fd65d7d6-99mfz\" (UID: \"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9\") " pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.461997 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.461897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9-tls-certs\") pod \"model-serving-api-fd65d7d6-99mfz\" (UID: \"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9\") " pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.463793 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.463772 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-h7qgp"] Jul 16 21:58:30.563233 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.563197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38dc02f0-a777-42f5-b070-dc312881a51c-cert\") pod \"odh-model-controller-84b54b7f97-h7qgp\" (UID: \"38dc02f0-a777-42f5-b070-dc312881a51c\") " pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.563407 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.563253 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqnf8\" (UniqueName: \"kubernetes.io/projected/f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9-kube-api-access-pqnf8\") pod \"model-serving-api-fd65d7d6-99mfz\" (UID: \"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9\") " pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.563407 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.563305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9-tls-certs\") pod \"model-serving-api-fd65d7d6-99mfz\" (UID: \"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9\") " pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.563407 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.563339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd465\" (UniqueName: \"kubernetes.io/projected/38dc02f0-a777-42f5-b070-dc312881a51c-kube-api-access-cd465\") pod \"odh-model-controller-84b54b7f97-h7qgp\" (UID: \"38dc02f0-a777-42f5-b070-dc312881a51c\") " pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.565636 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.565620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9-tls-certs\") pod \"model-serving-api-fd65d7d6-99mfz\" (UID: \"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9\") " pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.580322 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.580294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqnf8\" (UniqueName: \"kubernetes.io/projected/f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9-kube-api-access-pqnf8\") pod \"model-serving-api-fd65d7d6-99mfz\" (UID: \"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9\") " pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.664391 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.664368 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38dc02f0-a777-42f5-b070-dc312881a51c-cert\") pod \"odh-model-controller-84b54b7f97-h7qgp\" (UID: \"38dc02f0-a777-42f5-b070-dc312881a51c\") " pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.664500 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.664411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd465\" (UniqueName: \"kubernetes.io/projected/38dc02f0-a777-42f5-b070-dc312881a51c-kube-api-access-cd465\") pod \"odh-model-controller-84b54b7f97-h7qgp\" (UID: \"38dc02f0-a777-42f5-b070-dc312881a51c\") " pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.666484 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.666431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38dc02f0-a777-42f5-b070-dc312881a51c-cert\") pod \"odh-model-controller-84b54b7f97-h7qgp\" (UID: \"38dc02f0-a777-42f5-b070-dc312881a51c\") " pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.677353 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.677335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd465\" (UniqueName: \"kubernetes.io/projected/38dc02f0-a777-42f5-b070-dc312881a51c-kube-api-access-cd465\") pod \"odh-model-controller-84b54b7f97-h7qgp\" (UID: \"38dc02f0-a777-42f5-b070-dc312881a51c\") " pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.727188 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.727165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:30.750132 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.750107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:30.857695 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.857660 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-99mfz"] Jul 16 21:58:30.860857 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:58:30.860828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dfd6b2_2e44_4d07_a6c4_6acb0403aef9.slice/crio-5554916035bd71d9ba141c90a571282938d34ea5bcaf664bf3237152975326b0 WatchSource:0}: Error finding container 5554916035bd71d9ba141c90a571282938d34ea5bcaf664bf3237152975326b0: Status 404 returned error can't find the container with id 5554916035bd71d9ba141c90a571282938d34ea5bcaf664bf3237152975326b0 Jul 16 21:58:30.883091 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:30.883069 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-h7qgp"] Jul 16 21:58:30.885726 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:58:30.885700 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dc02f0_a777_42f5_b070_dc312881a51c.slice/crio-d57d51ef14e3859ba8045a21389d77400b648d3d149978f0ba58c8b1d5fc2a5f WatchSource:0}: Error finding container d57d51ef14e3859ba8045a21389d77400b648d3d149978f0ba58c8b1d5fc2a5f: Status 404 returned error can't find the container with id d57d51ef14e3859ba8045a21389d77400b648d3d149978f0ba58c8b1d5fc2a5f Jul 16 21:58:31.473094 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:31.473037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" event={"ID":"38dc02f0-a777-42f5-b070-dc312881a51c","Type":"ContainerStarted","Data":"d57d51ef14e3859ba8045a21389d77400b648d3d149978f0ba58c8b1d5fc2a5f"} Jul 16 21:58:31.474478 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:31.474450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-99mfz" event={"ID":"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9","Type":"ContainerStarted","Data":"5554916035bd71d9ba141c90a571282938d34ea5bcaf664bf3237152975326b0"} Jul 16 21:58:33.484779 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:33.484739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-99mfz" event={"ID":"f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9","Type":"ContainerStarted","Data":"f644896b04a4ba9ccf02eeada4615269bda17236758c2e6956c0ae533540a878"} Jul 16 21:58:33.485236 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:33.484904 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:33.507000 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:33.506947 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-99mfz" podStartSLOduration=1.364300571 podStartE2EDuration="3.506915222s" totalImagesPullingTime="2.142614651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:33.506577927 +0000 UTC m=+525.368152690" watchObservedRunningTime="2026-07-16 21:58:33.506915222 +0000 UTC m=+525.368489983" Jul 16 21:58:35.494211 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:35.494168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" event={"ID":"38dc02f0-a777-42f5-b070-dc312881a51c","Type":"ContainerStarted","Data":"3820b0a4d645858fe3e4af3cffd088c0354556711aa3f12a296bc027412aae60"} Jul 16 21:58:35.494677 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:35.494307 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:35.513216 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:35.513170 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" podStartSLOduration=1.48105866 podStartE2EDuration="5.513154746s" totalImagesPullingTime="4.032096086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:35.512085985 +0000 UTC m=+527.373660750" watchObservedRunningTime="2026-07-16 21:58:35.513154746 +0000 UTC m=+527.374729507" Jul 16 21:58:44.491860 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:44.491783 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-99mfz" Jul 16 21:58:46.500795 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:46.500768 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-h7qgp" Jul 16 21:58:47.508332 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.508287 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-mk2nx"] Jul 16 21:58:47.511503 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.511486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.515387 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.515366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 16 21:58:47.515387 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.515379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8pwwf\"" Jul 16 21:58:47.522209 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.522185 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mk2nx"] Jul 16 21:58:47.591483 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.591455 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b507aabd-3d89-4e29-ae48-d168e6ea309f-model-cache\") pod \"s3-init-mk2nx\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.591483 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.591487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfv8d\" (UniqueName: \"kubernetes.io/projected/b507aabd-3d89-4e29-ae48-d168e6ea309f-kube-api-access-mfv8d\") pod \"s3-init-mk2nx\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.692310 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.692276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b507aabd-3d89-4e29-ae48-d168e6ea309f-model-cache\") pod \"s3-init-mk2nx\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.692449 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.692315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfv8d\" (UniqueName: \"kubernetes.io/projected/b507aabd-3d89-4e29-ae48-d168e6ea309f-kube-api-access-mfv8d\") pod \"s3-init-mk2nx\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.692641 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.692615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b507aabd-3d89-4e29-ae48-d168e6ea309f-model-cache\") pod \"s3-init-mk2nx\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.703874 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.703855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfv8d\" (UniqueName: \"kubernetes.io/projected/b507aabd-3d89-4e29-ae48-d168e6ea309f-kube-api-access-mfv8d\") pod \"s3-init-mk2nx\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.820080 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.820000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mk2nx" Jul 16 21:58:47.972371 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:47.972339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-mk2nx"] Jul 16 21:58:47.975555 ip-10-0-128-250 kubenswrapper[2576]: W0716 21:58:47.975526 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb507aabd_3d89_4e29_ae48_d168e6ea309f.slice/crio-c27346e291062ea03ab355115d8ca5b0dc3b9c8c56554562c46ea52de5cf0387 WatchSource:0}: Error finding container c27346e291062ea03ab355115d8ca5b0dc3b9c8c56554562c46ea52de5cf0387: Status 404 returned error can't find the container with id c27346e291062ea03ab355115d8ca5b0dc3b9c8c56554562c46ea52de5cf0387 Jul 16 21:58:48.542616 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:48.542576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mk2nx" event={"ID":"b507aabd-3d89-4e29-ae48-d168e6ea309f","Type":"ContainerStarted","Data":"c27346e291062ea03ab355115d8ca5b0dc3b9c8c56554562c46ea52de5cf0387"} Jul 16 21:58:51.561368 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:51.561330 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mk2nx" event={"ID":"b507aabd-3d89-4e29-ae48-d168e6ea309f","Type":"ContainerStarted","Data":"b9fd78aea505d602b168df5d166bd6756eca5e9f5da309232575c954389c9c2d"} Jul 16 21:58:52.566328 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:52.566293 2576 generic.go:356] "Generic (PLEG): container finished" podID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerID="b9fd78aea505d602b168df5d166bd6756eca5e9f5da309232575c954389c9c2d" exitCode=0 Jul 16 21:58:52.566802 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:52.566366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mk2nx" event={"ID":"b507aabd-3d89-4e29-ae48-d168e6ea309f","Type":"ContainerDied","Data":"b9fd78aea505d602b168df5d166bd6756eca5e9f5da309232575c954389c9c2d"} Jul 16 21:58:56.847226 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:56.847206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jul 16 21:58:57.589592 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:57.589554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mk2nx" event={"ID":"b507aabd-3d89-4e29-ae48-d168e6ea309f","Type":"ContainerStarted","Data":"4319226e07e0cdb70b326e55db28905d0f08731156a5b727641030575cd898d0"} Jul 16 21:58:57.607783 ip-10-0-128-250 kubenswrapper[2576]: I0716 21:58:57.607733 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-mk2nx" podStartSLOduration=2.310440639 podStartE2EDuration="10.607718571s" totalImagesPullingTime="7.025463213s" totalInitContainerRuntime="1.271814719s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:57.607007254 +0000 UTC m=+549.468582016" watchObservedRunningTime="2026-07-16 21:58:57.607718571 +0000 UTC m=+549.469293332" Jul 16 22:00:02.810587 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:02.810550 2576 generic.go:356] "Generic (PLEG): container finished" podID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerID="4319226e07e0cdb70b326e55db28905d0f08731156a5b727641030575cd898d0" exitCode=0 Jul 16 22:00:02.811023 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:02.810620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mk2nx" event={"ID":"b507aabd-3d89-4e29-ae48-d168e6ea309f","Type":"ContainerDied","Data":"4319226e07e0cdb70b326e55db28905d0f08731156a5b727641030575cd898d0"} Jul 16 22:00:03.930790 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:03.930767 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mk2nx" Jul 16 22:00:04.052913 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.052871 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfv8d\" (UniqueName: \"kubernetes.io/projected/b507aabd-3d89-4e29-ae48-d168e6ea309f-kube-api-access-mfv8d\") pod \"b507aabd-3d89-4e29-ae48-d168e6ea309f\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " Jul 16 22:00:04.052913 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.052916 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b507aabd-3d89-4e29-ae48-d168e6ea309f-model-cache\") pod \"b507aabd-3d89-4e29-ae48-d168e6ea309f\" (UID: \"b507aabd-3d89-4e29-ae48-d168e6ea309f\") " Jul 16 22:00:04.055033 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.054997 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b507aabd-3d89-4e29-ae48-d168e6ea309f-kube-api-access-mfv8d" (OuterVolumeSpecName: "kube-api-access-mfv8d") pod "b507aabd-3d89-4e29-ae48-d168e6ea309f" (UID: "b507aabd-3d89-4e29-ae48-d168e6ea309f"). InnerVolumeSpecName "kube-api-access-mfv8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:00:04.072625 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.072568 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b507aabd-3d89-4e29-ae48-d168e6ea309f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b507aabd-3d89-4e29-ae48-d168e6ea309f" (UID: "b507aabd-3d89-4e29-ae48-d168e6ea309f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:00:04.154029 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.154007 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfv8d\" (UniqueName: \"kubernetes.io/projected/b507aabd-3d89-4e29-ae48-d168e6ea309f-kube-api-access-mfv8d\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:00:04.154029 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.154028 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b507aabd-3d89-4e29-ae48-d168e6ea309f-model-cache\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:00:04.817871 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.817844 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-mk2nx" Jul 16 22:00:04.817871 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.817863 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-mk2nx" event={"ID":"b507aabd-3d89-4e29-ae48-d168e6ea309f","Type":"ContainerDied","Data":"c27346e291062ea03ab355115d8ca5b0dc3b9c8c56554562c46ea52de5cf0387"} Jul 16 22:00:04.818046 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:04.817895 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27346e291062ea03ab355115d8ca5b0dc3b9c8c56554562c46ea52de5cf0387" Jul 16 22:00:17.026514 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.026478 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg"] Jul 16 22:00:17.026892 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.026721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerName="download-hf-model" Jul 16 22:00:17.026892 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.026731 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerName="download-hf-model" Jul 16 22:00:17.026892 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.026740 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerName="s3-init" Jul 16 22:00:17.026892 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.026746 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerName="s3-init" Jul 16 22:00:17.026892 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.026791 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b507aabd-3d89-4e29-ae48-d168e6ea309f" containerName="s3-init" Jul 16 22:00:17.029700 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.029683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.035161 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.035141 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" Jul 16 22:00:17.035469 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.035451 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 16 22:00:17.035538 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.035470 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" Jul 16 22:00:17.035538 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.035530 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jul 16 22:00:17.037896 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.037875 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\"" Jul 16 22:00:17.038194 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.038178 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 16 22:00:17.038271 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.038202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jul 16 22:00:17.083313 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.083286 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg"] Jul 16 22:00:17.151363 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.151330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96eeb52b-4743-47a6-84b5-c206d12018f5-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.151519 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.151376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.151519 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.151412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.151519 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.151438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qdt\" (UniqueName: \"kubernetes.io/projected/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-api-access-t2qdt\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.151519 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.151498 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96eeb52b-4743-47a6-84b5-c206d12018f5-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.252941 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.252886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253124 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.252987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253124 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.253026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qdt\" (UniqueName: \"kubernetes.io/projected/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-api-access-t2qdt\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253124 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.253092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96eeb52b-4743-47a6-84b5-c206d12018f5-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253315 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.253294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96eeb52b-4743-47a6-84b5-c206d12018f5-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253663 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.253640 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253768 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.253671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96eeb52b-4743-47a6-84b5-c206d12018f5-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.253847 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.253830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.255535 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.255513 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96eeb52b-4743-47a6-84b5-c206d12018f5-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.280298 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.280237 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qdt\" (UniqueName: \"kubernetes.io/projected/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-api-access-t2qdt\") pod \"isvc-sklearn-graph-1-predictor-57c485b568-mrdqg\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.339158 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.339129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:17.481961 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.481867 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg"] Jul 16 22:00:17.485307 ip-10-0-128-250 kubenswrapper[2576]: W0716 22:00:17.485278 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96eeb52b_4743_47a6_84b5_c206d12018f5.slice/crio-97a066a26a4bc172a87f270d2b0f7174f8cf56d82be3bd49f78f287e5add6e3f WatchSource:0}: Error finding container 97a066a26a4bc172a87f270d2b0f7174f8cf56d82be3bd49f78f287e5add6e3f: Status 404 returned error can't find the container with id 97a066a26a4bc172a87f270d2b0f7174f8cf56d82be3bd49f78f287e5add6e3f Jul 16 22:00:17.520276 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.520116 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh"] Jul 16 22:00:17.524543 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.524525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.527667 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.527642 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Jul 16 22:00:17.527855 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.527839 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Jul 16 22:00:17.535184 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.535160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh"] Jul 16 22:00:17.646619 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.646588 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6"] Jul 16 22:00:17.649786 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.649765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.653144 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.653116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Jul 16 22:00:17.653273 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.653250 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Jul 16 22:00:17.655722 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.655704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zkr\" (UniqueName: \"kubernetes.io/projected/406a6ace-a702-4e4c-8f45-66442558ff36-kube-api-access-z6zkr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.655817 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.655730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406a6ace-a702-4e4c-8f45-66442558ff36-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.655817 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.655759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.655817 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.655785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/406a6ace-a702-4e4c-8f45-66442558ff36-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.656082 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.655877 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.662916 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.662894 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6"] Jul 16 22:00:17.757136 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.757136 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcs2\" (UniqueName: \"kubernetes.io/projected/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-api-access-fdcs2\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zkr\" (UniqueName: \"kubernetes.io/projected/406a6ace-a702-4e4c-8f45-66442558ff36-kube-api-access-z6zkr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757183 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406a6ace-a702-4e4c-8f45-66442558ff36-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757208 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757252 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/406a6ace-a702-4e4c-8f45-66442558ff36-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.757353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.757745 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406a6ace-a702-4e4c-8f45-66442558ff36-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.757855 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.758007 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.757987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.759871 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.759845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/406a6ace-a702-4e4c-8f45-66442558ff36-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.767035 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.767011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zkr\" (UniqueName: \"kubernetes.io/projected/406a6ace-a702-4e4c-8f45-66442558ff36-kube-api-access-z6zkr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-ptfhh\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.838975 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.838890 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:17.858117 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcs2\" (UniqueName: \"kubernetes.io/projected/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-api-access-fdcs2\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.858229 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.858229 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.858353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.858353 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.858763 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.858998 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.858979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.859090 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.859037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.860887 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.860861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.864657 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.864628 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerStarted","Data":"8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25"} Jul 16 22:00:17.864761 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.864701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerStarted","Data":"97a066a26a4bc172a87f270d2b0f7174f8cf56d82be3bd49f78f287e5add6e3f"} Jul 16 22:00:17.867481 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.867458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcs2\" (UniqueName: \"kubernetes.io/projected/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-api-access-fdcs2\") pod \"isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.959326 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.959289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:17.978780 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:17.978717 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh"] Jul 16 22:00:17.980887 ip-10-0-128-250 kubenswrapper[2576]: W0716 22:00:17.980849 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406a6ace_a702_4e4c_8f45_66442558ff36.slice/crio-b313c7176c2e2fdd0e76b2dc03113068f4a9f2ff4bb8fbc12614e55666612418 WatchSource:0}: Error finding container b313c7176c2e2fdd0e76b2dc03113068f4a9f2ff4bb8fbc12614e55666612418: Status 404 returned error can't find the container with id b313c7176c2e2fdd0e76b2dc03113068f4a9f2ff4bb8fbc12614e55666612418 Jul 16 22:00:18.104291 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:18.104264 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6"] Jul 16 22:00:18.106135 ip-10-0-128-250 kubenswrapper[2576]: W0716 22:00:18.106105 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2b6cce_e7ef_4b78_a032_5c2657ba8592.slice/crio-0fc0e6567ce89fd6382e5f63f6d1a4f1dfef4d447cfc85528cc5be6aa2b20e6a WatchSource:0}: Error finding container 0fc0e6567ce89fd6382e5f63f6d1a4f1dfef4d447cfc85528cc5be6aa2b20e6a: Status 404 returned error can't find the container with id 0fc0e6567ce89fd6382e5f63f6d1a4f1dfef4d447cfc85528cc5be6aa2b20e6a Jul 16 22:00:18.869689 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:18.869651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerStarted","Data":"a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f"} Jul 16 22:00:18.869875 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:18.869718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerStarted","Data":"b313c7176c2e2fdd0e76b2dc03113068f4a9f2ff4bb8fbc12614e55666612418"} Jul 16 22:00:18.871167 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:18.871133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerStarted","Data":"33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746"} Jul 16 22:00:18.871279 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:18.871192 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerStarted","Data":"0fc0e6567ce89fd6382e5f63f6d1a4f1dfef4d447cfc85528cc5be6aa2b20e6a"} Jul 16 22:00:21.882962 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:21.882911 2576 generic.go:356] "Generic (PLEG): container finished" podID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerID="8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25" exitCode=0 Jul 16 22:00:21.883336 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:21.882957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerDied","Data":"8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25"} Jul 16 22:00:21.884687 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:21.884617 2576 generic.go:356] "Generic (PLEG): container finished" podID="406a6ace-a702-4e4c-8f45-66442558ff36" containerID="a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f" exitCode=0 Jul 16 22:00:21.884753 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:21.884697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerDied","Data":"a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f"} Jul 16 22:00:21.886588 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:21.886532 2576 generic.go:356] "Generic (PLEG): container finished" podID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerID="33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746" exitCode=0 Jul 16 22:00:21.886646 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:21.886616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerDied","Data":"33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746"} Jul 16 22:00:50.067448 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:50.067406 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerStarted","Data":"a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3"} Jul 16 22:00:50.069827 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:50.069659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerStarted","Data":"bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e"} Jul 16 22:00:50.091949 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:50.085418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerStarted","Data":"0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e"} Jul 16 22:00:53.108876 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.108844 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerStarted","Data":"48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3"} Jul 16 22:00:53.109309 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.109106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:53.109309 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.109253 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:53.110850 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.110823 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:00:53.111171 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.111148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerStarted","Data":"bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed"} Jul 16 22:00:53.111343 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.111320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:53.111411 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.111366 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:53.112285 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.112256 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:00:53.113323 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.113304 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerStarted","Data":"7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4"} Jul 16 22:00:53.113459 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.113444 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:53.130845 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.130800 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podStartSLOduration=1.816619748 podStartE2EDuration="36.130787885s" totalImagesPullingTime="30.476446916s" totalInitContainerRuntime="3.837721221s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:53.12942119 +0000 UTC m=+664.990995954" watchObservedRunningTime="2026-07-16 22:00:53.130787885 +0000 UTC m=+664.992362646" Jul 16 22:00:53.151063 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.151023 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podStartSLOduration=2.002383833 podStartE2EDuration="36.151010579s" totalImagesPullingTime="30.504192205s" totalInitContainerRuntime="3.644434541s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:53.150330504 +0000 UTC m=+665.011905262" watchObservedRunningTime="2026-07-16 22:00:53.151010579 +0000 UTC m=+665.012585340" Jul 16 22:00:53.171445 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:53.171395 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podStartSLOduration=1.986347445 podStartE2EDuration="36.171382188s" totalImagesPullingTime="30.52435761s" totalInitContainerRuntime="3.660677133s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:53.170079597 +0000 UTC m=+665.031654359" watchObservedRunningTime="2026-07-16 22:00:53.171382188 +0000 UTC m=+665.032956950" Jul 16 22:00:54.117400 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:54.117349 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:00:54.117853 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:54.117457 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:00:54.117853 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:54.117701 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:00:54.118532 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:54.118510 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:00:55.121805 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:55.121760 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:00:59.121455 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:59.121428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:00:59.121964 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:59.121870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:00:59.121964 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:59.121879 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:00:59.122356 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:00:59.122334 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:01:00.126269 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:00.126241 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:01:00.126848 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:00.126820 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:01:09.121902 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:09.121861 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:01:09.122389 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:09.122299 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:01:10.127285 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:10.127242 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:01:19.122429 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:19.122387 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:01:19.122893 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:19.122388 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:01:20.127088 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:20.127044 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:01:29.122426 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:29.122387 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:01:29.122899 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:29.122389 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:01:30.126867 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:30.126827 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:01:39.122483 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:39.122441 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:01:39.122483 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:39.122458 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:01:40.126998 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:40.126958 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:01:49.121995 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:49.121892 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:01:49.122943 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:49.122907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:01:50.127497 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:50.127456 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:01:59.122385 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:01:59.122343 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused" Jul 16 22:02:00.127662 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:00.127626 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:02:05.680088 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:05.680058 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:02:08.684034 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:08.684003 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:02:37.216777 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.216736 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg"] Jul 16 22:02:37.217349 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.217065 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" containerID="cri-o://a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3" gracePeriod=30 Jul 16 22:02:37.217349 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.217116 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kube-rbac-proxy" containerID="cri-o://48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3" gracePeriod=30 Jul 16 22:02:37.307463 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.307427 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh"] Jul 16 22:02:37.307674 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.307652 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" containerID="cri-o://bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e" gracePeriod=30 Jul 16 22:02:37.307776 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.307741 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kube-rbac-proxy" containerID="cri-o://bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed" gracePeriod=30 Jul 16 22:02:37.449393 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.449362 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6"] Jul 16 22:02:37.449615 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.449594 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" containerID="cri-o://0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e" gracePeriod=30 Jul 16 22:02:37.449729 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.449702 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kube-rbac-proxy" containerID="cri-o://7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4" gracePeriod=30 Jul 16 22:02:37.489429 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.489404 2576 generic.go:356] "Generic (PLEG): container finished" podID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerID="48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3" exitCode=2 Jul 16 22:02:37.489568 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.489490 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerDied","Data":"48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3"} Jul 16 22:02:37.491864 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.491839 2576 generic.go:356] "Generic (PLEG): container finished" podID="406a6ace-a702-4e4c-8f45-66442558ff36" containerID="bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed" exitCode=2 Jul 16 22:02:37.491990 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:37.491921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerDied","Data":"bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed"} Jul 16 22:02:38.497266 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:38.497234 2576 generic.go:356] "Generic (PLEG): container finished" podID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerID="7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4" exitCode=2 Jul 16 22:02:38.497641 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:38.497312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerDied","Data":"7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4"} Jul 16 22:02:38.682207 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:38.682147 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Jul 16 22:02:39.118488 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:39.118451 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused" Jul 16 22:02:39.118659 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:39.118451 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Jul 16 22:02:39.123198 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:39.123176 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Jul 16 22:02:40.122026 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:40.121981 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused" Jul 16 22:02:41.446718 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.446694 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:02:41.509152 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.509111 2576 generic.go:356] "Generic (PLEG): container finished" podID="406a6ace-a702-4e4c-8f45-66442558ff36" containerID="bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e" exitCode=0 Jul 16 22:02:41.509314 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.509176 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" Jul 16 22:02:41.509314 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.509199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerDied","Data":"bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e"} Jul 16 22:02:41.509314 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.509260 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh" event={"ID":"406a6ace-a702-4e4c-8f45-66442558ff36","Type":"ContainerDied","Data":"b313c7176c2e2fdd0e76b2dc03113068f4a9f2ff4bb8fbc12614e55666612418"} Jul 16 22:02:41.509314 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.509287 2576 scope.go:117] "RemoveContainer" containerID="bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed" Jul 16 22:02:41.516774 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.516758 2576 scope.go:117] "RemoveContainer" containerID="bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e" Jul 16 22:02:41.523605 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.523580 2576 scope.go:117] "RemoveContainer" containerID="a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f" Jul 16 22:02:41.530092 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.530078 2576 scope.go:117] "RemoveContainer" containerID="bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed" Jul 16 22:02:41.530335 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:41.530317 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed\": container with ID starting with bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed not found: ID does not exist" containerID="bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed" Jul 16 22:02:41.530380 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.530344 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed"} err="failed to get container status \"bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed\": rpc error: code = NotFound desc = could not find container \"bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed\": container with ID starting with bf3f6d4c87b08d64bbe855937dc427c45b5c219dbfbc56a4eee2e3327b40d7ed not found: ID does not exist" Jul 16 22:02:41.530380 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.530362 2576 scope.go:117] "RemoveContainer" containerID="bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e" Jul 16 22:02:41.530557 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:41.530545 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e\": container with ID starting with bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e not found: ID does not exist" containerID="bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e" Jul 16 22:02:41.530598 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.530562 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e"} err="failed to get container status \"bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e\": rpc error: code = NotFound desc = could not find container \"bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e\": container with ID starting with bbb80fb12020aef0e169a96e33e536aa3afa33cefc94a9c89b777bd67db3916e not found: ID does not exist" Jul 16 22:02:41.530598 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.530573 2576 scope.go:117] "RemoveContainer" containerID="a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f" Jul 16 22:02:41.530785 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:41.530768 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f\": container with ID starting with a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f not found: ID does not exist" containerID="a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f" Jul 16 22:02:41.530822 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.530792 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f"} err="failed to get container status \"a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f\": rpc error: code = NotFound desc = could not find container \"a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f\": container with ID starting with a91859244d32d87461ba61318904994bcc0a57c23853db14e367d9416f38643f not found: ID does not exist" Jul 16 22:02:41.564094 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564044 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406a6ace-a702-4e4c-8f45-66442558ff36-kserve-provision-location\") pod \"406a6ace-a702-4e4c-8f45-66442558ff36\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " Jul 16 22:02:41.564181 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564109 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-kube-rbac-proxy-sar-config\") pod \"406a6ace-a702-4e4c-8f45-66442558ff36\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " Jul 16 22:02:41.564181 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-cabundle-cert\") pod \"406a6ace-a702-4e4c-8f45-66442558ff36\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " Jul 16 22:02:41.564281 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564195 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6zkr\" (UniqueName: \"kubernetes.io/projected/406a6ace-a702-4e4c-8f45-66442558ff36-kube-api-access-z6zkr\") pod \"406a6ace-a702-4e4c-8f45-66442558ff36\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " Jul 16 22:02:41.564346 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/406a6ace-a702-4e4c-8f45-66442558ff36-proxy-tls\") pod \"406a6ace-a702-4e4c-8f45-66442558ff36\" (UID: \"406a6ace-a702-4e4c-8f45-66442558ff36\") " Jul 16 22:02:41.564402 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/406a6ace-a702-4e4c-8f45-66442558ff36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "406a6ace-a702-4e4c-8f45-66442558ff36" (UID: "406a6ace-a702-4e4c-8f45-66442558ff36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:02:41.564506 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564489 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/406a6ace-a702-4e4c-8f45-66442558ff36-kserve-provision-location\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:41.564506 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564495 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "406a6ace-a702-4e4c-8f45-66442558ff36" (UID: "406a6ace-a702-4e4c-8f45-66442558ff36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:41.564582 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.564500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "406a6ace-a702-4e4c-8f45-66442558ff36" (UID: "406a6ace-a702-4e4c-8f45-66442558ff36"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:41.566199 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.566178 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406a6ace-a702-4e4c-8f45-66442558ff36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "406a6ace-a702-4e4c-8f45-66442558ff36" (UID: "406a6ace-a702-4e4c-8f45-66442558ff36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:02:41.566307 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.566287 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406a6ace-a702-4e4c-8f45-66442558ff36-kube-api-access-z6zkr" (OuterVolumeSpecName: "kube-api-access-z6zkr") pod "406a6ace-a702-4e4c-8f45-66442558ff36" (UID: "406a6ace-a702-4e4c-8f45-66442558ff36"). InnerVolumeSpecName "kube-api-access-z6zkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:02:41.665805 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.665772 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/406a6ace-a702-4e4c-8f45-66442558ff36-proxy-tls\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:41.665805 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.665801 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:41.666000 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.665815 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/406a6ace-a702-4e4c-8f45-66442558ff36-cabundle-cert\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:41.666000 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.665828 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zkr\" (UniqueName: \"kubernetes.io/projected/406a6ace-a702-4e4c-8f45-66442558ff36-kube-api-access-z6zkr\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:41.835344 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.835313 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh"] Jul 16 22:02:41.838721 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:41.838699 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-ptfhh"] Jul 16 22:02:42.150044 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.150016 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:02:42.270239 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270210 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96eeb52b-4743-47a6-84b5-c206d12018f5-kserve-provision-location\") pod \"96eeb52b-4743-47a6-84b5-c206d12018f5\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " Jul 16 22:02:42.270338 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270260 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96eeb52b-4743-47a6-84b5-c206d12018f5-proxy-tls\") pod \"96eeb52b-4743-47a6-84b5-c206d12018f5\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " Jul 16 22:02:42.270338 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270301 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-cabundle-cert\") pod \"96eeb52b-4743-47a6-84b5-c206d12018f5\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " Jul 16 22:02:42.270432 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2qdt\" (UniqueName: \"kubernetes.io/projected/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-api-access-t2qdt\") pod \"96eeb52b-4743-47a6-84b5-c206d12018f5\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " Jul 16 22:02:42.270488 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-rbac-proxy-sar-config\") pod \"96eeb52b-4743-47a6-84b5-c206d12018f5\" (UID: \"96eeb52b-4743-47a6-84b5-c206d12018f5\") " Jul 16 22:02:42.270549 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270527 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96eeb52b-4743-47a6-84b5-c206d12018f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96eeb52b-4743-47a6-84b5-c206d12018f5" (UID: "96eeb52b-4743-47a6-84b5-c206d12018f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:02:42.270754 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270720 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "96eeb52b-4743-47a6-84b5-c206d12018f5" (UID: "96eeb52b-4743-47a6-84b5-c206d12018f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:42.270754 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270730 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96eeb52b-4743-47a6-84b5-c206d12018f5-kserve-provision-location\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.270951 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.270901 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96eeb52b-4743-47a6-84b5-c206d12018f5" (UID: "96eeb52b-4743-47a6-84b5-c206d12018f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:42.272372 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.272345 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-api-access-t2qdt" (OuterVolumeSpecName: "kube-api-access-t2qdt") pod "96eeb52b-4743-47a6-84b5-c206d12018f5" (UID: "96eeb52b-4743-47a6-84b5-c206d12018f5"). InnerVolumeSpecName "kube-api-access-t2qdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:02:42.272549 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.272529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96eeb52b-4743-47a6-84b5-c206d12018f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96eeb52b-4743-47a6-84b5-c206d12018f5" (UID: "96eeb52b-4743-47a6-84b5-c206d12018f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:02:42.371216 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.371193 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96eeb52b-4743-47a6-84b5-c206d12018f5-proxy-tls\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.371216 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.371217 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-cabundle-cert\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.371332 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.371226 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qdt\" (UniqueName: \"kubernetes.io/projected/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-api-access-t2qdt\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.371332 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.371237 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96eeb52b-4743-47a6-84b5-c206d12018f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.382297 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.382274 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:02:42.471871 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.471845 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kserve-provision-location\") pod \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " Jul 16 22:02:42.472249 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.471878 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-proxy-tls\") pod \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " Jul 16 22:02:42.472249 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.471907 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdcs2\" (UniqueName: \"kubernetes.io/projected/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-api-access-fdcs2\") pod \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " Jul 16 22:02:42.472249 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.472030 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-cabundle-cert\") pod \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " Jul 16 22:02:42.472249 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.472064 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-rbac-proxy-sar-config\") pod \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\" (UID: \"3e2b6cce-e7ef-4b78-a032-5c2657ba8592\") " Jul 16 22:02:42.472249 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.472156 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e2b6cce-e7ef-4b78-a032-5c2657ba8592" (UID: "3e2b6cce-e7ef-4b78-a032-5c2657ba8592"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:02:42.472520 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.472323 2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kserve-provision-location\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.472520 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.472373 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e2b6cce-e7ef-4b78-a032-5c2657ba8592" (UID: "3e2b6cce-e7ef-4b78-a032-5c2657ba8592"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:42.472520 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.472426 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e2b6cce-e7ef-4b78-a032-5c2657ba8592" (UID: "3e2b6cce-e7ef-4b78-a032-5c2657ba8592"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 22:02:42.473781 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.473762 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e2b6cce-e7ef-4b78-a032-5c2657ba8592" (UID: "3e2b6cce-e7ef-4b78-a032-5c2657ba8592"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 22:02:42.473866 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.473828 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-api-access-fdcs2" (OuterVolumeSpecName: "kube-api-access-fdcs2") pod "3e2b6cce-e7ef-4b78-a032-5c2657ba8592" (UID: "3e2b6cce-e7ef-4b78-a032-5c2657ba8592"). InnerVolumeSpecName "kube-api-access-fdcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:02:42.514947 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.514896 2576 generic.go:356] "Generic (PLEG): container finished" podID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerID="0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e" exitCode=0 Jul 16 22:02:42.515051 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.514971 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" Jul 16 22:02:42.515051 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.514984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerDied","Data":"0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e"} Jul 16 22:02:42.515051 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.515048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6" event={"ID":"3e2b6cce-e7ef-4b78-a032-5c2657ba8592","Type":"ContainerDied","Data":"0fc0e6567ce89fd6382e5f63f6d1a4f1dfef4d447cfc85528cc5be6aa2b20e6a"} Jul 16 22:02:42.515205 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.515064 2576 scope.go:117] "RemoveContainer" containerID="7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4" Jul 16 22:02:42.517068 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.517047 2576 generic.go:356] "Generic (PLEG): container finished" podID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerID="a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3" exitCode=0 Jul 16 22:02:42.517177 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.517111 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" Jul 16 22:02:42.517177 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.517122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerDied","Data":"a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3"} Jul 16 22:02:42.517177 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.517165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg" event={"ID":"96eeb52b-4743-47a6-84b5-c206d12018f5","Type":"ContainerDied","Data":"97a066a26a4bc172a87f270d2b0f7174f8cf56d82be3bd49f78f287e5add6e3f"} Jul 16 22:02:42.523152 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.523122 2576 scope.go:117] "RemoveContainer" containerID="0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e" Jul 16 22:02:42.530490 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.530476 2576 scope.go:117] "RemoveContainer" containerID="33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746" Jul 16 22:02:42.537292 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.537274 2576 scope.go:117] "RemoveContainer" containerID="7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4" Jul 16 22:02:42.537533 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:42.537515 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4\": container with ID starting with 7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4 not found: ID does not exist" containerID="7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4" Jul 16 22:02:42.537605 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.537542 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4"} err="failed to get container status \"7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4\": rpc error: code = NotFound desc = could not find container \"7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4\": container with ID starting with 7464ca501bd805e61a03f2e2a822e2153517e5d36802ee4ec4858c9ef8b557d4 not found: ID does not exist" Jul 16 22:02:42.537605 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.537559 2576 scope.go:117] "RemoveContainer" containerID="0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e" Jul 16 22:02:42.537826 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:42.537777 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e\": container with ID starting with 0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e not found: ID does not exist" containerID="0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e" Jul 16 22:02:42.537897 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.537836 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e"} err="failed to get container status \"0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e\": rpc error: code = NotFound desc = could not find container \"0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e\": container with ID starting with 0c134ea2501b39755fc78f27732b5f1914a70071d15c58236b5c1a9d3a02ff0e not found: ID does not exist" Jul 16 22:02:42.537897 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.537858 2576 scope.go:117] "RemoveContainer" containerID="33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746" Jul 16 22:02:42.538152 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:42.538133 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746\": container with ID starting with 33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746 not found: ID does not exist" containerID="33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746" Jul 16 22:02:42.538218 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.538156 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746"} err="failed to get container status \"33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746\": rpc error: code = NotFound desc = could not find container \"33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746\": container with ID starting with 33c1e6af0385549ac7ffacd0655575a0ea7585410f00e7132fc7178155279746 not found: ID does not exist" Jul 16 22:02:42.538218 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.538184 2576 scope.go:117] "RemoveContainer" containerID="48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3" Jul 16 22:02:42.538718 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.538698 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6"] Jul 16 22:02:42.542350 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.542329 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-779dbb5fd4-kvfc6"] Jul 16 22:02:42.545117 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.545103 2576 scope.go:117] "RemoveContainer" containerID="a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3" Jul 16 22:02:42.551505 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.551488 2576 scope.go:117] "RemoveContainer" containerID="8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25" Jul 16 22:02:42.553994 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.553975 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg"] Jul 16 22:02:42.558013 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.557993 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-57c485b568-mrdqg"] Jul 16 22:02:42.558644 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.558625 2576 scope.go:117] "RemoveContainer" containerID="48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3" Jul 16 22:02:42.558875 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:42.558857 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3\": container with ID starting with 48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3 not found: ID does not exist" containerID="48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3" Jul 16 22:02:42.558918 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.558883 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3"} err="failed to get container status \"48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3\": rpc error: code = NotFound desc = could not find container \"48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3\": container with ID starting with 48f9d26c8a0e93f8be9438fe24f9ab8afefa57ebc37de874af5e422a133216b3 not found: ID does not exist" Jul 16 22:02:42.558918 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.558899 2576 scope.go:117] "RemoveContainer" containerID="a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3" Jul 16 22:02:42.559154 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:42.559139 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3\": container with ID starting with a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3 not found: ID does not exist" containerID="a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3" Jul 16 22:02:42.559194 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.559158 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3"} err="failed to get container status \"a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3\": rpc error: code = NotFound desc = could not find container \"a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3\": container with ID starting with a159a4b27557ca55cfeefacc710c75cce9b4c0aecb03f159d724d110f1055ce3 not found: ID does not exist" Jul 16 22:02:42.559194 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.559171 2576 scope.go:117] "RemoveContainer" containerID="8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25" Jul 16 22:02:42.559412 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:02:42.559395 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25\": container with ID starting with 8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25 not found: ID does not exist" containerID="8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25" Jul 16 22:02:42.559460 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.559416 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25"} err="failed to get container status \"8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25\": rpc error: code = NotFound desc = could not find container \"8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25\": container with ID starting with 8019df5865ed11e52d21d122bb10b63756b0b409397274cb4d897d1591f30e25 not found: ID does not exist" Jul 16 22:02:42.572729 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.572712 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-proxy-tls\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.572775 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.572732 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdcs2\" (UniqueName: \"kubernetes.io/projected/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-api-access-fdcs2\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.572775 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.572741 2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-cabundle-cert\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.572775 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.572750 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2b6cce-e7ef-4b78-a032-5c2657ba8592-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:02:42.683187 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.683119 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" path="/var/lib/kubelet/pods/3e2b6cce-e7ef-4b78-a032-5c2657ba8592/volumes" Jul 16 22:02:42.683610 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.683597 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" path="/var/lib/kubelet/pods/406a6ace-a702-4e4c-8f45-66442558ff36/volumes" Jul 16 22:02:42.684313 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:02:42.684298 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" path="/var/lib/kubelet/pods/96eeb52b-4743-47a6-84b5-c206d12018f5/volumes" Jul 16 22:40:08.334612 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334578 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rcpjl/must-gather-97nn2"] Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334820 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334830 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334840 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334847 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334855 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334860 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334868 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="storage-initializer" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334873 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="storage-initializer" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334880 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334885 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334893 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334898 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334906 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="storage-initializer" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334911 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="storage-initializer" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334918 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="storage-initializer" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334923 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="storage-initializer" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334943 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334949 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334989 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.334997 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.335004 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.335010 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="406a6ace-a702-4e4c-8f45-66442558ff36" containerName="kube-rbac-proxy" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.335015 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2b6cce-e7ef-4b78-a032-5c2657ba8592" containerName="kserve-container" Jul 16 22:40:08.337017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.335020 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="96eeb52b-4743-47a6-84b5-c206d12018f5" containerName="kube-rbac-proxy" Jul 16 22:40:08.337824 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.337810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.342854 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.342832 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rcpjl\"/\"kube-root-ca.crt\"" Jul 16 22:40:08.343003 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.342855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rcpjl\"/\"default-dockercfg-rmnpn\"" Jul 16 22:40:08.343003 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.342832 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rcpjl\"/\"openshift-service-ca.crt\"" Jul 16 22:40:08.354886 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.354864 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcpjl/must-gather-97nn2"] Jul 16 22:40:08.429490 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.429451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6wp\" (UniqueName: \"kubernetes.io/projected/cda8ab31-65c1-488b-adfa-2cab6222ccd2-kube-api-access-pl6wp\") pod \"must-gather-97nn2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.429672 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.429531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cda8ab31-65c1-488b-adfa-2cab6222ccd2-must-gather-output\") pod \"must-gather-97nn2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.530773 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.530741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6wp\" (UniqueName: \"kubernetes.io/projected/cda8ab31-65c1-488b-adfa-2cab6222ccd2-kube-api-access-pl6wp\") pod \"must-gather-97nn2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.530978 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.530796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cda8ab31-65c1-488b-adfa-2cab6222ccd2-must-gather-output\") pod \"must-gather-97nn2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.531101 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.531084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cda8ab31-65c1-488b-adfa-2cab6222ccd2-must-gather-output\") pod \"must-gather-97nn2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.539992 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.539972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6wp\" (UniqueName: \"kubernetes.io/projected/cda8ab31-65c1-488b-adfa-2cab6222ccd2-kube-api-access-pl6wp\") pod \"must-gather-97nn2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.646284 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.646198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:08.775182 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.775143 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rcpjl/must-gather-97nn2"] Jul 16 22:40:08.782728 ip-10-0-128-250 kubenswrapper[2576]: W0716 22:40:08.782684 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcda8ab31_65c1_488b_adfa_2cab6222ccd2.slice/crio-0500cc7ae3272c435a6af91e388d96f39cf18de9bd2471c63169abde69f5bb25 WatchSource:0}: Error finding container 0500cc7ae3272c435a6af91e388d96f39cf18de9bd2471c63169abde69f5bb25: Status 404 returned error can't find the container with id 0500cc7ae3272c435a6af91e388d96f39cf18de9bd2471c63169abde69f5bb25 Jul 16 22:40:08.784440 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.784423 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 22:40:08.911146 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:08.911059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcpjl/must-gather-97nn2" event={"ID":"cda8ab31-65c1-488b-adfa-2cab6222ccd2","Type":"ContainerStarted","Data":"0500cc7ae3272c435a6af91e388d96f39cf18de9bd2471c63169abde69f5bb25"} Jul 16 22:40:15.942578 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:15.942533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcpjl/must-gather-97nn2" event={"ID":"cda8ab31-65c1-488b-adfa-2cab6222ccd2","Type":"ContainerStarted","Data":"3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45"} Jul 16 22:40:15.942974 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:15.942592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcpjl/must-gather-97nn2" event={"ID":"cda8ab31-65c1-488b-adfa-2cab6222ccd2","Type":"ContainerStarted","Data":"1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8"} Jul 16 22:40:39.034776 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:39.034742 2576 generic.go:356] "Generic (PLEG): container finished" podID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerID="1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8" exitCode=0 Jul 16 22:40:39.034776 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:39.034779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rcpjl/must-gather-97nn2" event={"ID":"cda8ab31-65c1-488b-adfa-2cab6222ccd2","Type":"ContainerDied","Data":"1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8"} Jul 16 22:40:39.035259 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:39.035078 2576 scope.go:117] "RemoveContainer" containerID="1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8" Jul 16 22:40:39.420238 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:39.420163 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rcpjl_must-gather-97nn2_cda8ab31-65c1-488b-adfa-2cab6222ccd2/gather/0.log" Jul 16 22:40:43.823430 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:43.823339 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qjjxj_f051cc1e-0917-46a4-bf69-e69da8dd4b65/global-pull-secret-syncer/0.log" Jul 16 22:40:43.964748 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:43.964721 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dv7l5_a812acaa-a375-486d-a551-d7e2e7c8b391/konnectivity-agent/0.log" Jul 16 22:40:43.994308 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:43.994277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-250.ec2.internal_6750ee95623d52ab8fe01710cc128243/haproxy/0.log" Jul 16 22:40:44.969778 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:44.969748 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rcpjl/must-gather-97nn2"] Jul 16 22:40:44.970177 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:44.969942 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rcpjl/must-gather-97nn2" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="copy" containerID="cri-o://3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45" gracePeriod=2 Jul 16 22:40:44.978017 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:44.977993 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rcpjl/must-gather-97nn2"] Jul 16 22:40:44.982823 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:40:44.982794 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"cda8ab31-65c1-488b-adfa-2cab6222ccd2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-rcpjl\"/\"must-gather-97nn2\": pods \"must-gather-97nn2\" not found" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:45.196532 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.196509 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rcpjl_must-gather-97nn2_cda8ab31-65c1-488b-adfa-2cab6222ccd2/copy/0.log" Jul 16 22:40:45.199042 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.199025 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:45.201614 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:40:45.201592 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-97nn2\" is forbidden: User \"system:node:ip-10-0-128-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rcpjl\": no relationship found between node 'ip-10-0-128-250.ec2.internal' and this object" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:45.314112 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.314088 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl6wp\" (UniqueName: \"kubernetes.io/projected/cda8ab31-65c1-488b-adfa-2cab6222ccd2-kube-api-access-pl6wp\") pod \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " Jul 16 22:40:45.314227 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.314122 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cda8ab31-65c1-488b-adfa-2cab6222ccd2-must-gather-output\") pod \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\" (UID: \"cda8ab31-65c1-488b-adfa-2cab6222ccd2\") " Jul 16 22:40:45.315709 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.315686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda8ab31-65c1-488b-adfa-2cab6222ccd2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cda8ab31-65c1-488b-adfa-2cab6222ccd2" (UID: "cda8ab31-65c1-488b-adfa-2cab6222ccd2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 22:40:45.316189 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.316167 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda8ab31-65c1-488b-adfa-2cab6222ccd2-kube-api-access-pl6wp" (OuterVolumeSpecName: "kube-api-access-pl6wp") pod "cda8ab31-65c1-488b-adfa-2cab6222ccd2" (UID: "cda8ab31-65c1-488b-adfa-2cab6222ccd2"). InnerVolumeSpecName "kube-api-access-pl6wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 22:40:45.414881 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.414855 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6wp\" (UniqueName: \"kubernetes.io/projected/cda8ab31-65c1-488b-adfa-2cab6222ccd2-kube-api-access-pl6wp\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:40:45.414881 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:45.414878 2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cda8ab31-65c1-488b-adfa-2cab6222ccd2-must-gather-output\") on node \"ip-10-0-128-250.ec2.internal\" DevicePath \"\"" Jul 16 22:40:46.058646 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.058618 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rcpjl_must-gather-97nn2_cda8ab31-65c1-488b-adfa-2cab6222ccd2/copy/0.log" Jul 16 22:40:46.059096 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.059038 2576 generic.go:356] "Generic (PLEG): container finished" podID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerID="3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45" exitCode=143 Jul 16 22:40:46.059096 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.059087 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:46.059186 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.059138 2576 scope.go:117] "RemoveContainer" containerID="3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45" Jul 16 22:40:46.061894 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:40:46.061871 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-97nn2\" is forbidden: User \"system:node:ip-10-0-128-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rcpjl\": no relationship found between node 'ip-10-0-128-250.ec2.internal' and this object" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:46.067025 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.066849 2576 scope.go:117] "RemoveContainer" containerID="1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8" Jul 16 22:40:46.069735 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:40:46.069710 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-97nn2\" is forbidden: User \"system:node:ip-10-0-128-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rcpjl\": no relationship found between node 'ip-10-0-128-250.ec2.internal' and this object" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" pod="openshift-must-gather-rcpjl/must-gather-97nn2" Jul 16 22:40:46.078702 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.078684 2576 scope.go:117] "RemoveContainer" containerID="3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45" Jul 16 22:40:46.078963 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:40:46.078943 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45\": container with ID starting with 3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45 not found: ID does not exist" containerID="3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45" Jul 16 22:40:46.079008 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.078972 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45"} err="failed to get container status \"3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45\": rpc error: code = NotFound desc = could not find container \"3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45\": container with ID starting with 3986ce5cd5daf01be572e88eb572234bac8ee99753c1d264f1ba027f518c6b45 not found: ID does not exist" Jul 16 22:40:46.079008 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.078990 2576 scope.go:117] "RemoveContainer" containerID="1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8" Jul 16 22:40:46.079231 ip-10-0-128-250 kubenswrapper[2576]: E0716 22:40:46.079212 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8\": container with ID starting with 1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8 not found: ID does not exist" containerID="1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8" Jul 16 22:40:46.079271 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.079238 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8"} err="failed to get container status \"1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8\": rpc error: code = NotFound desc = could not find container \"1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8\": container with ID starting with 1248b62ef6286008caf79400d4471caf23d02a8814ce795dff1982fc57c39db8 not found: ID does not exist" Jul 16 22:40:46.684347 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:46.684314 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" path="/var/lib/kubelet/pods/cda8ab31-65c1-488b-adfa-2cab6222ccd2/volumes" Jul 16 22:40:48.605232 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:48.605207 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-79z9k_4c712b2f-3057-4593-a4f2-4fdc48eccd36/node-exporter/0.log" Jul 16 22:40:48.632786 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:48.632767 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-79z9k_4c712b2f-3057-4593-a4f2-4fdc48eccd36/kube-rbac-proxy/0.log" Jul 16 22:40:48.667588 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:48.667565 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-79z9k_4c712b2f-3057-4593-a4f2-4fdc48eccd36/init-textfile/0.log" Jul 16 22:40:51.934984 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.934948 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl"] Jul 16 22:40:51.935355 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.935178 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="gather" Jul 16 22:40:51.935355 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.935189 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="gather" Jul 16 22:40:51.935355 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.935209 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="copy" Jul 16 22:40:51.935355 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.935215 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="copy" Jul 16 22:40:51.935355 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.935254 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="gather" Jul 16 22:40:51.935355 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.935263 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda8ab31-65c1-488b-adfa-2cab6222ccd2" containerName="copy" Jul 16 22:40:51.940200 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.940181 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:51.943622 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.943600 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jnrjl\"/\"kube-root-ca.crt\"" Jul 16 22:40:51.943853 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.943838 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jnrjl\"/\"openshift-service-ca.crt\"" Jul 16 22:40:51.944571 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.944555 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jnrjl\"/\"default-dockercfg-gv8jf\"" Jul 16 22:40:51.951507 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:51.951484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl"] Jul 16 22:40:52.063316 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.063280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ht94\" (UniqueName: \"kubernetes.io/projected/57e88a51-c613-4ba2-863c-c39fde2d6acc-kube-api-access-6ht94\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.063517 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.063333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-lib-modules\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.063517 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.063427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-proc\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.063517 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.063474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-sys\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.063517 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.063491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-podres\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164052 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-proc\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164052 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-sys\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164240 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-podres\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164240 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ht94\" (UniqueName: \"kubernetes.io/projected/57e88a51-c613-4ba2-863c-c39fde2d6acc-kube-api-access-6ht94\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164240 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-sys\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164240 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-proc\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164240 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-lib-modules\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164390 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-podres\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.164390 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.164290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57e88a51-c613-4ba2-863c-c39fde2d6acc-lib-modules\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.174174 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.174151 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ht94\" (UniqueName: \"kubernetes.io/projected/57e88a51-c613-4ba2-863c-c39fde2d6acc-kube-api-access-6ht94\") pod \"perf-node-gather-daemonset-5mqfl\" (UID: \"57e88a51-c613-4ba2-863c-c39fde2d6acc\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.250155 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.250124 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:52.377074 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:52.377039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl"] Jul 16 22:40:52.380161 ip-10-0-128-250 kubenswrapper[2576]: W0716 22:40:52.380133 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57e88a51_c613_4ba2_863c_c39fde2d6acc.slice/crio-bc597d16a34d091c63e45eec4512f51a338748c5ed21328069a66962a61c15f5 WatchSource:0}: Error finding container bc597d16a34d091c63e45eec4512f51a338748c5ed21328069a66962a61c15f5: Status 404 returned error can't find the container with id bc597d16a34d091c63e45eec4512f51a338748c5ed21328069a66962a61c15f5 Jul 16 22:40:53.084946 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:53.084888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" event={"ID":"57e88a51-c613-4ba2-863c-c39fde2d6acc","Type":"ContainerStarted","Data":"136acce22bca1f7f119dd3b7a967fde407a035332c249e40bbf5e3fa0bc261b4"} Jul 16 22:40:53.085310 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:53.084969 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" event={"ID":"57e88a51-c613-4ba2-863c-c39fde2d6acc","Type":"ContainerStarted","Data":"bc597d16a34d091c63e45eec4512f51a338748c5ed21328069a66962a61c15f5"} Jul 16 22:40:53.085310 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:53.084997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:53.104153 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:53.104033 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" podStartSLOduration=2.104015941 podStartE2EDuration="2.104015941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:53.103562566 +0000 UTC m=+3064.965137326" watchObservedRunningTime="2026-07-16 22:40:53.104015941 +0000 UTC m=+3064.965590703" Jul 16 22:40:53.948363 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:53.948333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-588zz_66e702f9-c995-4223-99c2-900fa238ff8a/dns/0.log" Jul 16 22:40:53.974582 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:53.974559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-588zz_66e702f9-c995-4223-99c2-900fa238ff8a/kube-rbac-proxy/0.log" Jul 16 22:40:54.175049 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:54.175014 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dtt29_bc099ed7-cc93-4665-9d00-35bc13b47a3c/dns-node-resolver/0.log" Jul 16 22:40:54.872325 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:54.872295 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lt998_e4d190f3-cbee-4efd-b727-fd11a101fc52/node-ca/0.log" Jul 16 22:40:56.206180 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:56.206151 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tw46w_9fdd52a8-81dd-4146-a49e-207380169305/serve-healthcheck-canary/0.log" Jul 16 22:40:57.090880 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:57.090845 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qvrlv_53ca1d22-b598-4c7c-8f8f-9f675f9dbff8/kube-rbac-proxy/0.log" Jul 16 22:40:57.141258 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:57.141223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qvrlv_53ca1d22-b598-4c7c-8f8f-9f675f9dbff8/exporter/0.log" Jul 16 22:40:57.181801 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:57.181781 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qvrlv_53ca1d22-b598-4c7c-8f8f-9f675f9dbff8/extractor/0.log" Jul 16 22:40:59.097814 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:59.097785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-5mqfl" Jul 16 22:40:59.523528 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:59.523502 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-mhhzq_f4810e2b-10d8-4260-b62f-765bd9cdcddb/manager/0.log" Jul 16 22:40:59.556506 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:59.556480 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-99mfz_f6dfd6b2-2e44-4d07-a6c4-6acb0403aef9/server/0.log" Jul 16 22:40:59.654828 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:59.654787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-h7qgp_38dc02f0-a777-42f5-b070-dc312881a51c/manager/0.log" Jul 16 22:40:59.710063 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:59.710042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mk2nx_b507aabd-3d89-4e29-ae48-d168e6ea309f/s3-init/0.log" Jul 16 22:40:59.743924 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:40:59.743902 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-mk2nx_b507aabd-3d89-4e29-ae48-d168e6ea309f/download-hf-model/0.log" Jul 16 22:41:06.472662 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.472633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/kube-multus-additional-cni-plugins/0.log" Jul 16 22:41:06.503751 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.503724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/egress-router-binary-copy/0.log" Jul 16 22:41:06.530782 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.530754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/cni-plugins/0.log" Jul 16 22:41:06.558199 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.558168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/bond-cni-plugin/0.log" Jul 16 22:41:06.593039 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.593013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/routeoverride-cni/0.log" Jul 16 22:41:06.622707 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.622684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/whereabouts-cni-bincopy/0.log" Jul 16 22:41:06.660763 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.660739 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-s9v8c_3a46827d-b431-4e78-90d1-f0cdf47dff3c/whereabouts-cni/0.log" Jul 16 22:41:06.949142 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.949115 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t9r2v_5835d2d8-4b2e-47fe-813f-dd649c70d3fe/kube-multus/0.log" Jul 16 22:41:06.984121 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:06.984097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-27sgl_5914b657-8a03-4f03-90d5-b94f1bb5301a/network-metrics-daemon/0.log" Jul 16 22:41:07.018146 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:07.018120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-27sgl_5914b657-8a03-4f03-90d5-b94f1bb5301a/kube-rbac-proxy/0.log" Jul 16 22:41:08.011330 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.011301 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/ovn-controller/0.log" Jul 16 22:41:08.039810 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.039778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/ovn-acl-logging/0.log" Jul 16 22:41:08.069948 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.069902 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/kube-rbac-proxy-node/0.log" Jul 16 22:41:08.113065 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.113039 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/kube-rbac-proxy-ovn-metrics/0.log" Jul 16 22:41:08.154156 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.154123 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/northd/0.log" Jul 16 22:41:08.194668 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.194642 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/nbdb/0.log" Jul 16 22:41:08.224095 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.224073 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/sbdb/0.log" Jul 16 22:41:08.271374 ip-10-0-128-250 kubenswrapper[2576]: I0716 22:41:08.271289 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-564jb_25591d72-2bba-45c8-b70e-6c9b0eaf4142/ovnkube-controller/0.log"